critical thinking and evaluating information

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Working with sources
• What Is Critical Thinking? | Definition & Examples

What Is Critical Thinking? | Definition & Examples

Published on May 30, 2022 by Eoghan Ryan . Revised on May 31, 2023.

Critical thinking is the ability to effectively analyze information and form a judgment .

To think critically, you must be aware of your own biases and assumptions when encountering information, and apply consistent standards when evaluating sources .

Critical thinking skills help you to:

• Identify credible sources
• Evaluate and respond to arguments
• Assess alternative viewpoints
• Test hypotheses against relevant criteria

Table of contents

Why is critical thinking important, critical thinking examples, how to think critically, other interesting articles, frequently asked questions about critical thinking.

Critical thinking is important for making judgments about sources of information and forming your own arguments. It emphasizes a rational, objective, and self-aware approach that can help you to identify credible sources and strengthen your conclusions.

Critical thinking is important in all disciplines and throughout all stages of the research process . The types of evidence used in the sciences and in the humanities may differ, but critical thinking skills are relevant to both.

In academic writing , critical thinking can help you to determine whether a source:

• Is free from research bias
• Provides evidence to support its research findings
• Considers alternative viewpoints

Outside of academia, critical thinking goes hand in hand with information literacy to help you form opinions rationally and engage independently and critically with popular media.

Don't submit your assignments before you do this

The academic proofreading tool has been trained on 1000s of academic texts. Making it the most accurate and reliable proofreading tool for students. Free citation check included.

Try for free

Critical thinking can help you to identify reliable sources of information that you can cite in your research paper . It can also guide your own research methods and inform your own arguments.

Outside of academia, critical thinking can help you to be aware of both your own and others’ biases and assumptions.

Academic examples

However, when you compare the findings of the study with other current research, you determine that the results seem improbable. You analyze the paper again, consulting the sources it cites.

You notice that the research was funded by the pharmaceutical company that created the treatment. Because of this, you view its results skeptically and determine that more independent research is necessary to confirm or refute them. Example: Poor critical thinking in an academic context You’re researching a paper on the impact wireless technology has had on developing countries that previously did not have large-scale communications infrastructure. You read an article that seems to confirm your hypothesis: the impact is mainly positive. Rather than evaluating the research methodology, you accept the findings uncritically.

Nonacademic examples

However, you decide to compare this review article with consumer reviews on a different site. You find that these reviews are not as positive. Some customers have had problems installing the alarm, and some have noted that it activates for no apparent reason.

You revisit the original review article. You notice that the words “sponsored content” appear in small print under the article title. Based on this, you conclude that the review is advertising and is therefore not an unbiased source. Example: Poor critical thinking in a nonacademic context You support a candidate in an upcoming election. You visit an online news site affiliated with their political party and read an article that criticizes their opponent. The article claims that the opponent is inexperienced in politics. You accept this without evidence, because it fits your preconceptions about the opponent.

There is no single way to think critically. How you engage with information will depend on the type of source you’re using and the information you need.

However, you can engage with sources in a systematic and critical way by asking certain questions when you encounter information. Like the CRAAP test , these questions focus on the currency , relevance , authority , accuracy , and purpose of a source of information.

When encountering information, ask:

• Who is the author? Are they an expert in their field?
• What do they say? Is their argument clear? Can you summarize it?
• When did they say this? Is the source current?
• Where is the information published? Is it an academic article? Is it peer-reviewed ?
• Why did the author publish it? What is their motivation?
• How do they make their argument? Is it backed up by evidence? Does it rely on opinion, speculation, or appeals to emotion ? Do they address alternative arguments?

Critical thinking also involves being aware of your own biases, not only those of others. When you make an argument or draw your own conclusions, you can ask similar questions about your own writing:

• Am I only considering evidence that supports my preconceptions?
• Is my argument expressed clearly and backed up with credible sources?
• Would I be convinced by this argument coming from someone else?

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

• ChatGPT vs human editor
• ChatGPT citations
• Is ChatGPT trustworthy?
• Using ChatGPT for your studies
• What is ChatGPT?
• Chicago style
• Paraphrasing

 Plagiarism

• Types of plagiarism
• Self-plagiarism
• Avoiding plagiarism
• Academic integrity
• Consequences of plagiarism
• Common knowledge

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

• Missing commas and periods
• Incorrect usage of “et al.”
• Ampersands (&) in narrative citations
• Missing reference entries

Critical thinking refers to the ability to evaluate information and to be aware of biases or assumptions, including your own.

Like information literacy , it involves evaluating arguments, identifying and solving problems in an objective and systematic way, and clearly communicating your ideas.

Critical thinking skills include the ability to:

You can assess information and arguments critically by asking certain questions about the source. You can use the CRAAP test , focusing on the currency , relevance , authority , accuracy , and purpose of a source of information.

Ask questions such as:

• Who is the author? Are they an expert?
• How do they make their argument? Is it backed up by evidence?

A credible source should pass the CRAAP test  and follow these guidelines:

• The information should be up to date and current.
• The author and publication should be a trusted authority on the subject you are researching.
• The sources the author cited should be easy to find, clear, and unbiased.
• For a web source, the URL and layout should signify that it is trustworthy.

Information literacy refers to a broad range of skills, including the ability to find, evaluate, and use sources of information effectively.

Being information literate means that you:

• Know how to find credible sources
• Use relevant sources to inform your research
• Understand what constitutes plagiarism
• Know how to cite your sources correctly

Confirmation bias is the tendency to search, interpret, and recall information in a way that aligns with our pre-existing values, opinions, or beliefs. It refers to the ability to recollect information best when it amplifies what we already believe. Relatedly, we tend to forget information that contradicts our opinions.

Although selective recall is a component of confirmation bias, it should not be confused with recall bias.

On the other hand, recall bias refers to the differences in the ability between study participants to recall past events when self-reporting is used. This difference in accuracy or completeness of recollection is not related to beliefs or opinions. Rather, recall bias relates to other factors, such as the length of the recall period, age, and the characteristics of the disease under investigation.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Ryan, E. (2023, May 31). What Is Critical Thinking? | Definition & Examples. Scribbr. Retrieved September 4, 2024, from https://www.scribbr.com/working-with-sources/critical-thinking/

Is this article helpful?

Eoghan Ryan

Other students also liked, student guide: information literacy | meaning & examples, what are credible sources & how to spot them | examples, applying the craap test & evaluating sources, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

The Peak Performance Center

The pursuit of performance excellence, critical thinking.

Critical thinking refers to the process of actively analyzing, assessing, synthesizing, evaluating and reflecting on information gathered from observation, experience, or communication. It is thinking in a clear, logical, reasoned, and reflective manner to solve problems or make decisions. Basically, critical thinking is taking a hard look at something to understand what it really means.

Critical Thinkers

Critical thinkers do not simply accept all ideas, theories, and conclusions as facts. They have a mindset of questioning ideas and conclusions. They make reasoned judgments that are logical and well thought out by assessing the evidence that supports a specific theory or conclusion.

When presented with a new piece of new information, critical thinkers may ask questions such as;

“What information supports that?”

“How was this information obtained?”

“Who obtained the information?”

“How do we know the information is valid?”

“Why is it that way?”

“What makes it do that?”

“How do we know that?”

“Are there other possibilities?”

Combination of Analytical and Creative Thinking

Many people perceive critical thinking just as analytical thinking. However, critical thinking incorporates both analytical thinking and creative thinking. Critical thinking does involve breaking down information into parts and analyzing the parts in a logical, step-by-step manner. However, it also involves challenging consensus to formulate new creative ideas and generate innovative solutions. It is critical thinking that helps to evaluate and improve your creative ideas.

Elements of Critical Thinking

Critical thinking involves:

• Gathering relevant information
• Evaluating information
• Asking questions
• Assessing bias or unsubstantiated assumptions
• Making inferences from the information and filling in gaps
• Using abstract ideas to interpret information
• Formulating ideas
• Weighing opinions
• Reaching well-reasoned conclusions
• Considering alternative possibilities
• Testing conclusions
• Verifying if evidence/argument support the conclusions

Developing Critical Thinking Skills

Critical thinking is considered a higher order thinking skills, such as analysis, synthesis, deduction, inference, reason, and evaluation. In order to demonstrate critical thinking, you would need to develop skills in;

Interpreting : understanding the significance or meaning of information

Analyzing : breaking information down into its parts

Connecting : making connections between related items or pieces of information.

Integrating : connecting and combining information to better understand the relationship between the information.

Evaluating : judging the value, credibility, or strength of something

Reasoning : creating an argument through logical steps

Deducing : forming a logical opinion about something based on the information or evidence that is available

Inferring : figuring something out through reasoning based on assumptions and ideas

Generating : producing new information, ideas, products, or ways of viewing things.

Blooms Taxonomy

Bloom’s Taxonomy Revised

Mind Mapping

Chunking Information

Brainstorming

Copyright © 2024 | WordPress Theme by MH Themes

• Programs & Services
• Delphi Center

Ideas to Action (i2a)

• What is Critical Thinking?

The ability to think critically calls for a higher-order thinking than simply the ability to recall information.

Definitions of critical thinking, its elements, and its associated activities fill the educational literature of the past forty years. Critical thinking has been described as an ability to question; to acknowledge and test previously held assumptions; to recognize ambiguity; to examine, interpret, evaluate, reason, and reflect; to make informed judgments and decisions; and to clarify, articulate, and justify positions (Hullfish & Smith, 1961; Ennis, 1962; Ruggiero, 1975; Scriven, 1976; Hallet, 1984; Kitchener, 1986; Pascarella & Terenzini, 1991; Mines et al., 1990; Halpern, 1996; Paul & Elder, 2001; Petress, 2004; Holyoak & Morrison, 2005; among others).

After a careful review of the mountainous body of literature defining critical thinking and its elements, UofL has chosen to adopt the language of Michael Scriven and Richard Paul (2003) as a comprehensive, concise operating definition:

Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action.

Paul and Scriven go on to suggest that critical thinking is based on: "universal intellectual values that transcend subject matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness. It entails the examination of those structures or elements of thought implicit in all reasoning: purpose, problem, or question-at-issue, assumptions, concepts, empirical grounding; reasoning leading to conclusions, implication and consequences, objections from alternative viewpoints, and frame of reference. Critical thinking - in being responsive to variable subject matter, issues, and purposes - is incorporated in a family of interwoven modes of thinking, among them: scientific thinking, mathematical thinking, historical thinking, anthropological thinking, economic thinking, moral thinking, and philosophical thinking."

This conceptualization of critical thinking has been refined and developed further by Richard Paul and Linder Elder into the Paul-Elder framework of critical thinking. Currently, this approach is one of the most widely published and cited frameworks in the critical thinking literature. According to the Paul-Elder framework, critical thinking is the:

• Analysis of thinking by focusing on the parts or structures of thinking ("the Elements of Thought")
• Evaluation of thinking by focusing on the quality ("the Universal Intellectual Standards")
• Improvement of thinking by using what you have learned ("the Intellectual Traits")

Selection of a Critical Thinking Framework

The University of Louisville chose the Paul-Elder model of Critical Thinking as the approach to guide our efforts in developing and enhancing our critical thinking curriculum. The Paul-Elder framework was selected based on criteria adapted from the characteristics of a good model of critical thinking developed at Surry Community College. The Paul-Elder critical thinking framework is comprehensive, uses discipline-neutral terminology, is applicable to all disciplines, defines specific cognitive skills including metacognition, and offers high quality resources.

Why the selection of a single critical thinking framework?

The use of a single critical thinking framework is an important aspect of institution-wide critical thinking initiatives (Paul and Nosich, 1993; Paul, 2004). According to this view, critical thinking instruction should not be relegated to one or two disciplines or departments with discipline specific language and conceptualizations. Rather, critical thinking instruction should be explicitly infused in all courses so that critical thinking skills can be developed and reinforced in student learning across the curriculum. The use of a common approach with a common language allows for a central organizer and for the development of critical thinking skill sets in all courses.

• SACS & QEP
• Planning and Implementation
• Why Focus on Critical Thinking?
• Paul-Elder Critical Thinking Framework
• Culminating Undergraduate Experience
• Community Engagement
• Frequently Asked Questions
• What is i2a?

Copyright © 2012 - University of Louisville , Delphi Center

Critical Thinking and Evaluating Information

• Introduction
• Words Of Wisdom

What Is Critical Thinking?

Five simple strategies to sharpen your critical thinking, were critical thinking skills used in this video.

• Critical Thinking and Reflective Judgement
• Problem Solving Skills
• Critical Reading
• Critical Writing
• Use the CRAAP Test
• Evaluating Fake News
• Explore Different Viewpoints
• The Peer-Review Process
• Critical ThinkingTutorials
• Books on Critical Thinking
• Explore More With These Links

Live Chat with a Librarian 24/7

Text a Librarian at 912-600-2782

Critical thinking is a complex process of deliberation that involves a wide range of skills and attitudes. It includes:

• identifying other people's positions,  arguments and conclusions 
• evaluating the evidence  for alternative points of view
• weighing up the opposing arguments  and evidence fairly
• being able to read between the lines,  seeing behind surfaces and identifying false or unfair assumptions
• recognizing techniques  used to make certain positions more appealing than others, such as false logic and persuasive devices
• reflecting on issues  in a structured way, bringing logic and insight to bear
• drawing conclusions  about whether arguments are valid and justifiable, based on good evidence and sensible assumptions
• presenting a point of view  in a structured, clear, well-reasoned way that convinces others

(Contrell, 2011)

Check Out More From This Source: 

A well-cultivated critical thinker:

• raises vital questions and problems, formulating them clearly and precisely;
• gathers and assesses relevant information, using abstract ideas to interpret it effectively  come to well-reasoned conclusions and solutions, testing them against relevant criteria and standards;
• thinks openmindedly within alternative systems of thought, recognizing and assessing, as need be, their assumptions, implications, and practical consequences; and
• communicates effectively with others in figuring out solutions to complex problems.

Critical thinking is, in short, self-directed, self-disciplined, self-monitored, and self-corrective thinking. It presupposes assent to rigorous standards of excellence and mindful command of their use. It entails effective communication and problem solving abilities and a commitment to overcome our native egocentrism and sociocentrism.  

(Taken from Richard Paul and Linda Elder,  The Miniature Guide to Critical Thinking Concepts and Tools,  Foundation for Critical Thinking Press, 2008)

Source: criticalthinking.org

Is the sky really blue? That might seem obvious. But sometimes things are more nuanced and complicated than you think. Here are five strategies to boost your critical thinking skills. Animated by Ana Stefaniak. Made in partnership with The Open University.

Video Source: BBC Ideas

Video Source: Dr. Bachyrycz

• << Previous: Introduction
• Next: Critical Thinking and Reflective Judgement >>
• Last Updated: Mar 25, 2024 8:18 AM
• URL: https://libguides.coastalpines.edu/thinkcritically

• Get started with computers
• Learn Microsoft Office
• Apply for a job
• Improve my work skills
• Design nice-looking docs
• Getting Started
• Smartphones & Tablets
• Typing Tutorial
• Online Learning
• Basic Internet Skills
• Online Safety
• Social Media
• Zoom Basics
• Google Docs
• Google Sheets
• Career Planning
• Resume Writing
• Cover Letters
• Job Search and Networking
• Business Communication
• Entrepreneurship 101
• Careers without College
• Job Hunt for Today
• 3D Printing
• Freelancing 101
• Personal Finance
• Sharing Economy
• Decision-Making
• Graphic Design
• Photography
• Image Editing
• Learning WordPress
• Language Learning
• Critical Thinking
• For Educators
• Translations
• Staff Picks
• English expand_more expand_less

Critical Thinking and Decision-Making  - What is Critical Thinking?

Critical thinking and decision-making  -, what is critical thinking, critical thinking and decision-making what is critical thinking.

Critical Thinking and Decision-Making: What is Critical Thinking?

Lesson 1: what is critical thinking, what is critical thinking.

Critical thinking is a term that gets thrown around a lot. You've probably heard it used often throughout the years whether it was in school, at work, or in everyday conversation. But when you stop to think about it, what exactly is critical thinking and how do you do it ?

Watch the video below to learn more about critical thinking.

Simply put, critical thinking is the act of deliberately analyzing information so that you can make better judgements and decisions . It involves using things like logic, reasoning, and creativity, to draw conclusions and generally understand things better.

This may sound like a pretty broad definition, and that's because critical thinking is a broad skill that can be applied to so many different situations. You can use it to prepare for a job interview, manage your time better, make decisions about purchasing things, and so much more.

The process

As humans, we are constantly thinking . It's something we can't turn off. But not all of it is critical thinking. No one thinks critically 100% of the time... that would be pretty exhausting! Instead, it's an intentional process , something that we consciously use when we're presented with difficult problems or important decisions.

Improving your critical thinking

In order to become a better critical thinker, it's important to ask questions when you're presented with a problem or decision, before jumping to any conclusions. You can start with simple ones like What do I currently know? and How do I know this? These can help to give you a better idea of what you're working with and, in some cases, simplify more complex issues.  

Real-world applications

Let's take a look at how we can use critical thinking to evaluate online information . Say a friend of yours posts a news article on social media and you're drawn to its headline. If you were to use your everyday automatic thinking, you might accept it as fact and move on. But if you were thinking critically, you would first analyze the available information and ask some questions :

• What's the source of this article?
• Is the headline potentially misleading?
• What are my friend's general beliefs?
• Do their beliefs inform why they might have shared this?

After analyzing all of this information, you can draw a conclusion about whether or not you think the article is trustworthy.

Critical thinking has a wide range of real-world applications . It can help you to make better decisions, become more hireable, and generally better understand the world around you.

/en/problem-solving-and-decision-making/why-is-it-so-hard-to-make-decisions/content/

• Study and research support
• Academic skills

Critical thinking

Evaluating information, critical reading questions.

You will need to make use of high-quality evidence and information. But with so much information out there, how can you decide what to use? Not everything you find, particularly on the web, is appropriate for academic study. You need to be able to think critically and judge what is relevant and appropriate for your purpose.

If we consider the critical thinking model, the description segment in particular helps us to generate the type of questions you may ask when making an initial evaluation of information.

Questions you may ask include:

• What is the source about? How relevant is to your context?
• Who is the author and are they subject experts?
• When was the material published? Is that significant to you? Has the information been superseded?
• What evidence is provided? Are there references that you could follow up to check any claims made?

Use our full set of critical thinking questions (PDF) to help you engage critically. These questions will take you through the description, analysis and evaluation stages as presented in the critical thinking model.

The questions will help you to evaluate the relevance and significance of your reading to your research or assignments.

You will be prompted to make the decision on how you will use the reading and what the relation is between this reading and the other information you have read.

However, this is not a comprehensive list and you may need to adapt or add your own questions for your subject, different assignments or a particular purpose.

Evaluating the author’s argument

When you have found a section of text that is directly relevant to your essay title or research, you need to slow down and read it more intensively. Critical reading is about analysing and evaluating the author's argument, not just looking for information.

The author should outline their viewpoint clearly and provide evidence from reliable sources to back this up.

Ask yourself:

• Is the author's argument clear? How is it presented?
• What evidence is provided? How is it used and interpreted?
• Is the argument convincing? How does it reach its conclusion?

You can also download our Evaluating information checklist (PDF) . This is just a guide; some of the questions will be more relevant to your context than others.

Making effective notes

Making meaningful notes as you read can help you to clarify your thinking, organise your ideas and engage critically with the information.

Microsoft OneNote can help you organise your notes. You can upload screenshots or scans of your journal article, textbook etc. and annotate them with your own comments. The “highlighter tool” can be particularly useful for noting strengths and weaknesses in an argument. Available as part of the Student Advantage Office 365 pack .

For advice on specialised software and tools that can help you with your studies and assessments (such as dictation, screen reading or mind mapping tools), visit the Disability Services Assistive Technology page. You can also find a wider range of personalised support, from academic adjustments and alternative exam arrangements to advice on extra funding. To find out more and register, contact Disability Services . 

Biosciences

• Finding recommended reading
• Journals and journal papers
• Industry news
• Reading and understanding

Thinking critically

• Referencing and plagiarism
• Creating and sharing
• Further support

Library Help

Your liaison librarian is oliver.

Oliver Allchin

Liaison Librarian

[email protected]

Your librarian can be contacted by email or you can book an online appointment .

It's important to think critically about the places you search for information and the information you find. The following tutorials and workshops will support you to develop a questioning approach to information and learn more about critical evaluation.

Thinking critically resources

• << Previous: Reading and understanding
• Next: Referencing and plagiarism >>
• Last Reviewed: Sep 5, 2024 12:00 PM
• URL: https://libraryguides.sheffield.ac.uk/guidebiosci

Critical thinking definition

Critical thinking, as described by Oxford Languages, is the objective analysis and evaluation of an issue in order to form a judgement.

Active and skillful approach, evaluation, assessment, synthesis, and/or evaluation of information obtained from, or made by, observation, knowledge, reflection, acumen or conversation, as a guide to belief and action, requires the critical thinking process, which is why it's often used in education and academics.

Some even may view it as a backbone of modern thought.

However, it's a skill, and skills must be trained and encouraged to be used at its full potential.

People turn up to various approaches in improving their critical thinking, like:

• Developing technical and problem-solving skills
• Engaging in more active listening
• Actively questioning their assumptions and beliefs
• Seeking out more diversity of thought
• Opening up their curiosity in an intellectual way etc.

Is critical thinking useful in writing?

Critical thinking can help in planning your paper and making it more concise, but it's not obvious at first. We carefully pinpointed some the questions you should ask yourself when boosting critical thinking in writing:

• What information should be included?
• Which information resources should the author look to?
• What degree of technical knowledge should the report assume its audience has?
• What is the most effective way to show information?
• How should the report be organized?
• How should it be designed?
• What tone and level of language difficulty should the document have?

Usage of critical thinking comes down not only to the outline of your paper, it also begs the question: How can we use critical thinking solving problems in our writing's topic?

Let's say, you have a Powerpoint on how critical thinking can reduce poverty in the United States. You'll primarily have to define critical thinking for the viewers, as well as use a lot of critical thinking questions and synonyms to get them to be familiar with your methods and start the thinking process behind it.

Are there any services that can help me use more critical thinking?

We understand that it's difficult to learn how to use critical thinking more effectively in just one article, but our service is here to help.

We are a team specializing in writing essays and other assignments for college students and all other types of customers who need a helping hand in its making. We cover a great range of topics, offer perfect quality work, always deliver on time and aim to leave our customers completely satisfied with what they ordered.

The ordering process is fully online, and it goes as follows:

• Select the topic and the deadline of your essay.
• Provide us with any details, requirements, statements that should be emphasized or particular parts of the essay writing process you struggle with.
• Leave the email address, where your completed order will be sent to.
• Select your prefered payment type, sit back and relax!

With lots of experience on the market, professionally degreed essay writers , online 24/7 customer support and incredibly low prices, you won't find a service offering a better deal than ours.

• Information Literacy

Critical Thinking

Critical thinking defined, assistive technologies, critical thinking & information literacy.

• Meta-cognitive Awareness Inventory

Research Help

Library Terms

Multilingual Glossary for Today’s Library Users - Definitions

Universal design for learning.

• Reference Information
• Find books and E-books
• Find articles
• Statistical Sources
• Evaluate Information
• Cite sources
• Writing Style Manuals
• Fake News and Information Literacy

Critical Thinking can be thought of in terms of

• Reasonable thinking
• Reflective thinking
• Evaluative thinking
• Mindful thought
• Intellectually-disciplined thought

Critical Thinking as Defined by the National Council for Excellence in Critical Thinking, 1987

A statement by Michael Scriven & Richard Paul, presented at the 8th Annual International Conference on Critical Thinking and Education Reform, Summer 1987.

Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience, reflection, reasoning, or communication, as a guide to belief and action. In its exemplary form, it is based on universal intellectual values that transcend subject matter divisions: clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.

• Critically Thinking

You’ve done some great work so far, thumbs up!  Now we are going to look at information access and evaluation, another important skill for your research skills toolbox.

Information has many facets, and it’s important to understand how these components contribute to writing your research paper. sometimes, you are looking for snippets of information that capture your thoughts or ideas. but when you  access and evaluate resources  you need to  think deeply  and  critically  about the  resource  you want to use to  support your argument  in your writing assignment., information  resources come in a  variety of formats, such as books, e-books, scholarly and peer reviewed articles, articles from trade magazines, newspapers, and, depending on your topic, streaming videos; audio files or blog posts .  but one thing they have in common is that they have  identifiable attributes  for you to consider. these attributes help you to determine if the resource is relevant to your topic., so what are these facets.

• The date the source  was published or created.
• If the article is not been published recently, you must ask yourself why you want to use it as a source. Is the material dated? Or does it offer some insight that warrants being cited (i.e., is it a classic in the field? a neglected contribution to the literature?)
• Is this part of a larger source?
• For example, is it a chapter in a book or e-book? Article in a journal or newspaper?
• What about that source tells you this?
• Article in a newspaper or trade magazine
• Book Review
• Scholarly and peer reviewed article

There are a number of questions you should ask of these different formats. If your source is from a periodical, is that source considered credible, for example, a major newspaper such as the  New York Times  or  Washington Post ? If your source is from a trade magazine, does it offer a skewed perspective, based on its position in industry or ideology? Does it show bias? If from a website, where does the site get its facts? Does it cite scholarly articles, clearly indicate its sources? Have other credible sources questioned its objectivity?

• What do you know about the author ? ( Where they   work ,  what they do ,  other sources  they’ve created, their relationship to the subject or topic?
• What else  might you find out about the author/s?

Once you identify these aspects, you need to ask some critical questions to evaluate your sources.

• What is your source about?   What is the author’s argument?  If you can’t tell from the information that’s been provided,  context or clues  within the source will help you make a reasonable guess.
• What would you say about the language  used in the source? Is it difficult to understand or fairly simple?
• Who do you think is the  audience  for your source? Why?
• What about the  visuals  in your source? For example, are the images used to  support the message ,  provide evidence , or  give you information about the author ? Are there  images  that  distract ?
• Remember that the PGCC Library Databases have been vetted by Teaching and Library faculty to ensure that the content meets the curriculum plan of the college. If you are using articles from a PGCC Library Database, you will never have to pay to access the article.

Now we are going to look at an article obtained from a library database about BLACKLIVESMATTER  and see if we can consider access and evaluation of the article based on the criteria above. This article is from PsycArticles, a PROQUEST database.

The article title is : “Participation in Black Lives Matter and deferred action for childhood arrivals: Modern activism among Black and Latino college students”

What do we know about the author/authors?

If you click on the Hyperlink for the Author’s name, you’ll find other articles that have been published by the author.

The article is published in the  Journal of Diversity in Higher Education.  If you click on the name of the Journal you will find out information about the journal. 

The 9.3 (Sep 2016): 203-215; indicates that it is Volume 9; Issue 3 dated September 2016: pages 203-215.

How can we tell that this is a scholarly and peer reviewed article? What components of this article indicate that this is a research paper?

If you look at these components, you will find that they meet the test of a scholarly and peer reviewed research article. The article uses technical terminology, and it follows a standard research format—it has an abstract, a review of the literature, methodology, results, conclusion, and references.

So, after looking at this article, you have concluded that this is a peer-reviewed research article. Next you’ll need to evaluate the source. You’ll want to consider  what this source is about . From reading the abstract above, can you  consider through what lens or perspective might this author be writing?

First, look at the language in the article. Is it  clear, concise  and  easily readable ? Based on the language, who do you think  the AUDIENCE  is for this source? Students? Researchers? Is it for the  average reader  or for someone who might want to write a research paper?

Now let’s look at the article’s presentation of data. You will find four tables that report on the study:  Study Variables by Race; BLM and DACA involvement by Race, Ethnicity and Gender; Average level of political activism; Predicting BLM and DACA Involvement . Do these tables help you understand the impact of study better? Why or Why not? 

Now let’s return to the language of the article and see if we can tell if this article  pro-BLACKLIVESMATTERS or not? How can you tell?  Are there  clues in how the abstract  is written that help you to  infer the author’s position ? For example, does this statement from the article give you a perspective as to the direction of the article, “ Two 21st century sociopolitical movements that have emerged to counteract racial/ethnic marginalization in the United States are BLM and advocacy for DACA legislation. BLM activists seek legislative changes to decrease the negative (and often life threatening) effects of discriminatory practices in our justice and political systems ”.

Your analysis of the author’s attitude involves you interpreting the article’s tone—in the preceding sentence, the author does not use language to undermine BLM—it doesn’t say “claims to” or “reportedly” or “seemingly” in describing the impact of the movement. It does not use charged political rhetoric to suggest BLM’s worsens marginalization or to undercut its assertions about the level of discrimination.

Then you have judge the usefulness of the source:

If you are writing about the  influence of the BLACKLIVESMATTER  movement and  activism , is this article  good for your paper ? Why or Why not?

Let’s look at the abstract, where the article claims that “ Political activism is one way racially/ethnically marginalized youth can combat institutional discrimination and seek legislative change toward equality and justice. In the current study, we examine participation in #BlackLivesMatter (BLM) and advocacy for Deferred Action for Childhood Arrivals (DACA) as political activism popular among youth.”

First, determine whether the article may provide evidence to support your argument. This involves paying close attention to the article’s thesis and to its supporting evidence. What  do you think the  article  is  saying overall ? What is the  takeaway ? How does it relate to your own argument? This involves considerable reflection on your part.

For example, does this statement argue your topic?  “Finally, scholars suggest that experiencing racial/ethnic discrimination likely contributes to greater participation in political activism as a mechanism to mitigate future instances of discrimination (Hope & Jagers, 2014; Hope & Spencer, in press)”. That really depends on what  your  thesis is. You may find that this conclusion is too broad, and you may then refine your own position. In an engagement with scholarly articles, you may be forced to think more clearly about your own position.

Secondly , you must determine how much research has been done on this topic.  Where does this article fit in the overall field of scholarship? You can’t simply assume that one article has vanquished all others from the field of intellectual battle. In this analysis, you must examine the article’s limitations:  What wasn’t included or  what was missing  from the article? Have you seen other articles that challenge the author’s perspective? Do you want—for example—to see evidence of political activism actually leading to change? Or is the article’s claim too weak? After all, the sentence above simply says it’s one way to seek change, not the most effective.

Remember, research is a process. You want to find the best scholarly articles not only to support your own claims, but to challenge your assumptions and help refine your conclusions. As we’ve seen, that involves determining whether an article appears in a respectable scholarly journal—as citing weak and unprofessional sources destroys your credibility and offers no real challenge. Instead, you should exercise your analytical and argumentative skills on the best scholarship available. 

Accessibility Masterlist

This MasterList, edited by Gregg Vanderheiden Ph.D.,   is designed to serve as a resource for researchers, developers, students, and others interested in understanding or developing products that incorporate one or more of these features.

Each feature or approach is then listed below along with applicable disabilities to each feature are marked with the following icons:

• B  - Blindness (For our purposes, blindness is defined as no or very low vision - such that text cannot be read at any magnification)
• LV  - Low Vision
• CLL  - Cognitive, Language, and Learning Disabilities (including low literacy)
• PHY  - Physical Disabilities
• D/HOH  - Deaf and Hard of Hearing
• American Sign Language Dictionary

Search and compare thousands of words and phrases in American Sign Language (ASL). The largest collection of free video signs online.

Braille Translator 

Brailletranslator.org is a simple way to convert text to braille notation. This supports nearly all Grade Two braille contractions.

• Braille Translator

Voyant Tools (Corpus Analysis)

Voyant Tools is an open-source, web-based application for performing text analysis. It supports scholarly reading and interpretation of texts or corpus, particularly by scholars in the digital humanities, but also by students and the general public. It can be used to analyze online texts or ones uploaded by users. (Source:  Wikipedia )

• Voyant Tools (Corpus Analysis)

Critical Thinking & Information Literacy - Parallel Processes

• Realize the task
• Explore, formulate, question, make connections
• Search and find
• Collect and organize
• Analyze, evaluate, interpret
• Apply understanding
• Communicate, present, share

Did you know that you can request a RESEARCH CONSULTATION appointment?  This is a one-on-one assistance with your research related need. 

Whether it's in-person, e-mail , phone, chat or text to 301-637-4609, you can ask a librarian for research help. 

It's important to understand library terms in order for you to do your research. If you have questions about the terminology used in the tutorial you can check this Glossary of Library Terms.

Abstract :  A summary or brief description of the content of another long work. An abstract is often provided along with the citation to a work.

Annotated bibliography:  a bibliography in which a brief explanatory or evaluate note is added to each reference or citation. An annotation can be helpful to the researcher in evaluating whether the source is relevant to a given topic or line of inquiry.

Archives : 1. A space which houses historical or public records. 2. The historical or public records themselves, which are generally non-circulating materials such as collections of personal papers, rare books, Ephemera, etc.

Article : A brief work—generally between 1 and 35 pages in length—on a topic. Often published as part of a journal, magazine, or newspaper.

Author : The person(s) or organization(s) that wrote or compiled a document. Looking for information under its author's name is one option in searching.

Bibliography : A list containing citations to the resources used in writing a research paper or other document. See also Reference.

Book : A relatively lengthy work, often on a single topic. May be in print or electronic.

Boolean operator : A word—such as AND, OR, or NOT—that commands a computer to combine search terms. Helps to narrow (AND, NOT) or broaden (OR) searches.

Call number : A group of letters and/or numbers that identifies a specific item in a library and provides a way for organizing library holdings. Three major types of call numbers are Dewey Decimal, Library of Congress, and Superintendent of Documents.

Catalog : A database (either online or on paper cards) listing and describing the books, journals, government documents, audiovisual and other materials held by a library. Various search terms allow you to look for items in the catalog.

Check-out : To borrow an item from a library for a fixed period of time in order to read, listen to, or view it. Check-out periods vary by library. Items are checked out at the circulation desk.

Circulation : The place in the library, often a desk, where you check out, renew, and return library materials. You may also place a hold, report an item missing from the shelves, or pay late fees or fines there.

Citation : A reference to a book, magazine or journal article, or other work containing all the information necessary to identify and locate that work. A citation to a book includes its author's name, title, publisher and place of publication, and date of publication.

Controlled vocabulary : Standardized terms used in searching a specific database.

Course reserve : Select books, articles, videotapes, or other materials that instructors want students to read or view for a particular course. These materials are usually kept in one area of the library and circulate for only a short period of time. See also Electronic reserve.

Descriptor : A word that describes the subject of an article or book; used in many computer databases.

Dissertation : An extended written treatment of a subject (like a book) submitted by a graduate student as a requirement for a doctorate.

DOI : Acronym for Digital Object Identifier. It is a unique alphanumeric string assigned by the publisher to a digital object.

E-book (or Electronic book) : An electronic version of a book that can be read on a computer or mobile device.

Editor : A person or group responsible for compiling the writings of others into a single information source. Looking for information under the editor's name is one option in searching.

Electronic reserve (or E-reserve) : An electronic version of a course reserve that is read on a computer display screen. See also Course reserve.

Encyclopedia : A work containing information on all branches of knowledge or treating comprehensively a particular branch of knowledge (such as history or chemistry). Often has entries or articles arranged alphabetically.

Hold : A request to have an item saved (put aside) to be picked up later. Holds can generally, be placed on any regularly circulating library material in-person or online.

Holdings : The materials owned by a library.

Index : 1. A list of names or topics—usually found at the end of a publication—that directs you to the pages where those names or topics are discussed within the publication. 2. A printed or electronic publication that provides references to periodical articles or books by their subject, author, or other search terms.

Interlibrary services/loan : A service that allows you to borrow materials from other libraries through your own library. See also Document delivery.

Journal : A publication, issued on a regular basis, which contains scholarly research published as articles, papers, research reports, or technical reports. See also Periodical.

Limits/limiters : Options used in searching that restrict your results to only information resources meeting certain other, non-subject-related, criteria. Limiting options vary by database, but common options include limiting results to materials available full-text in the database, to scholarly publications, to materials written in a particular language, to materials available in a particular location, or to materials published at a specific time. 

Magazine : A publication, issued on a regular basis, containing popular articles, written and illustrated in a less technical manner than the articles found in a journal.

Microform : A reduced sized photographic reproduction of printed information on reel to reel film (microfilm) or film cards (microfiche) or opaque pages that can be read with a microform reader/printer.

Newspaper : A publication containing information about varied topics that are pertinent to general information, a geographic area, or a specific subject matter (i.e. business, culture, education). Often published daily.

Online Public Access Catalog (OPAC) : A computerized database that can be searched in various ways— such as by keyword, author, title, subject, or call number— to find out what resources a library owns. OPAC’s will supply listings of the title, call number, author, location, and description of any items matching one's search. Also referred to as “library catalog ” or “online catalog.”

PDF : A file format developed by Adobe Acrobat® that allows files to be transmitted from one computer to another while retaining their original appearance both on-screen and when printed. An acronym for Portable Document Format.

Peer-reviewed journal : Peer review is a process by which editors have experts in a field review books or articles submitted for publication by the experts’ peers. Peer review helps to ensure the quality of an information source. A peer-reviewed journal is also called a refereed journal or scholarly journal.

Periodical : An information source published in multiple parts at regular intervals (daily, weekly, monthly, biannually). Journals, magazines, and newspapers are all periodicals. See also Serial.

Plagiarism : Using the words or ideas of others without acknowledging the original source.

Primary source : An original record of events, such as a diary, a newspaper article, a public record, or scientific documentation.

Print : The written symbols of a language as portrayed on paper. Information sources may be either print or electronic.

Publisher : An entity or company that produces and issues books, journals, newspapers, or other publications.

Recall : A request for the return of library material before the due date.

Refereed journal: See Peer-reviewed journal.

Reference : 1. A service that helps people find needed information. 2. Sometimes "reference" refers to reference collections, such as encyclopedias, indexes, handbooks, directories, etc. 3. A citation to a work is also known as a reference.

Renewal : An extension of the loan period for library materials.

Reserve : 1. A service providing special, often short-term, access to course-related materials (book or article readings, lecture notes, sample tests) or to other materials (CD-ROMs, audio-visual materials, current newspapers or magazines). 2. Also the physical location—often a service desk or room—within a library where materials on reserve are kept. Materials can also be made available electronically. See also Course reserve, Electronic reserve.

Scholarly journal : See Peer-reviewed journal.

Search statement/Search Query : Words entered into the search box of a database or search engine when looking for information. Words relating to an information source's author, editor, title, subject heading or keyword serve as search terms. Search terms can be combined by using Boolean operators and can also be used with limits/limiters.

Secondary sources : Materials such as books and journal articles that analyze primary sources. Secondary sources usually provide evaluation or interpretation of data or evidence found in original research or documents such as historical manuscripts or memoirs.

Serial : Publications such as journals, magazines, and newspapers that are generally published multiple times per year, month, or week. Serials usually have number volumes and issues.

Stacks : Shelves in the library where materials—typically books—are stored. Books in the stacks are normally arranged by call number. May be referred to as “book stacks.”

Style manual : An information source providing guidelines for people who are writing research papers. A style manual outlines specific formats for arranging research papers and citing the sources that are used in writing the paper.

Subject heading : Descriptions of an information source’s content assigned to make finding information easier. See also Controlled vocabulary, Descriptors.

Title : The name of a book, article, or other information sources. Upload: To transfer information from a computer system or a personal computer to another computer system or a larger computer system.

Virtual reference: A service allowing library users to ask questions through email, text message, or live-chat as opposed to coming to the reference desk at the library and asking a question in person. Also referred to as “online reference” or “e-reference.”

Multilingual Glossary for Today’s Library Users

If English is not your first language, then this resource will help you navigate the definitions of library terms in the following languages: English, Chinese, Korean, Japanese, French, Spanish, Arabic, and Vietnamese.

• Multilingual Glossary for Today’s Library Users - Definitions The Glossary provides terms an ESL speaker might find useful and a listing of the terms that are most likely to be used in a library.
• Multilingual Glossary for Today’s Library Users - Language Table Here is a list of definitions that you can also find the translation in English, Chinese, Korean, Japanese, French, Spanish, Arabic, and Vietnamese.

Universal Design for Learning (UDL)

• UDL Core Apps
• Universal Design for Learning Research Guide
• << Previous: Information Literacy
• Next: Reference Information >>
• Last Updated: Jul 20, 2023 1:57 PM
• URL: https://pgcc.libguides.com/c.php?g=631064

mySmowltech

Critical thinking: definition and how to improve its skills

Recruitment

Discover our proctoring plans

Receive an ebook on proctoring solutions. SMOWL is the most complete and customizable proctoring software.

Recent posts

HR technology trends: the future landscapes

Ghosting after interviews: tips and ideas to avoid it

Proctoring and sustainability: A Winning team

Critical thinking is based on the observation and analysis of facts and evidences to return rational, skeptical and unbiased judgments.   

This type of thinking involves a series of skills that can be created but also improved, as we will see throughout this article in which we will begin by defining the concept and end with tips to build and improve the skills related to critical thinking.

What is critical thinking?

Critical thinking is a discipline based on the ability of people to observe, elucidate and analyze information, facts and evidences in order to judge or decide if it is right or wrong.

It goes beyond mere curiosity, simple knowledge or analysis of any kind of fact or information.

People who develop this type of outlook are able to logically connect ideas and defend them with weighty opinions that ultimately help them make better decisions.

How to build and improve critical thinking skills?

Building and improving critical thinking skills involves focusing on a number of abilities and capacities .

To begin the critical thinking process all ideas must be open and all options must be understood as much as possible.

Even the dumbest or craziest idea can end up being the gateway to the most intelligent and successful conclusion.

The problem with having an open mind is that it is the most difficult path and often involves a greater challenge and effort. It is well known that the easy thing to do is to go with the obvious and the commonly accepted but this has no place in critical thinking.

By contrast, it is helpful not to make hasty decisions and to weigh the problem in its entirety after a first moment of awareness.

Finally, practicing active listening will help you to receive feedback from others and to understand other points of view that may help you as a reference.

Impartiality

An important point in the critical thinking process is the development of the ability to identify biases and maintain an impartial view in evaluations.

To improve this aspect it is advisable to have tools to be able to identify and recognize the prejudices and biases you have and try to leave them completely aside when thinking about the solution.

Subscribe today to SMOWL’s weekly newsletter!

Discover the latest trends in eLearning, technology, and innovation, alongside experts in assessment and talent management. Stay informed about industry updates and get the information you need.

Simply fill out the form and stay up-to-date with everything relevant in our field.

Observation

Observation allows you to see each and every detail , no matter how small, subtle or inconsequential they may be or seem to be.

Behind the superficial information hides a universe of data, sources and experiences that help you make the best decision.

One of the pillars of critical thinking is objectivity. This forces you to base your value judgments on established facts that you will have gathered after a correct research process. 

At this point in the process you should also be clear about the influencing factors to be taken into account and those that can be left out.

Remember that your research is not only about gathering a good amount of information that puts the maximum number of options, variables or situations on the table. 

For the information to be of quality, it must be based on reliable and trustworthy sources.

If the information you have to collect is based on the comments and opinions of third parties, try to exercise quality control but without interference. 

To do this, ask open-ended questions that bring all the nuances to the table and at the same time serve to sift out possible biases.

With the research process completed, it is time to analyze the sources and information gathered.

At this point, your analytical skills will help you to discard what does not conform to unconventional thinking, to prioritize among the information that is of value, to identify possible trends and to draw your own conclusions.

One of the skills that characterize a person with critical thinking is their ability to recognize patterns and connections between all the pieces of information they handle in their research.

This allows them to draw conclusions of great relevance on which to base their predictions with weighty foundations.

Analytical thinking is sometimes confused with critical thinking. The former only uses facts and data, while the latter incorporates other nuances such as emotions, experiences or opinions.

One of the problems with critical thinking is that it can be developed to infinity and beyond. You can always keep looking for new avenues of investigation and new lines of argument by stretching inference to limits that may not be necessary.

At this point it is important to clarify that inference is the process of drawing conclusions from initial premises or hypotheses.

Knowing when to stop the research and thinking process and move on to the next stage in which you put into practice the actions considered appropriate is necessary.

Communication

The information you collect in your research is not top secret material. On the contrary, your knowledge sharing with other people who are involved in the next steps of the process is so important.

Think that your analytical ability to extract the information and your conclusions can serve to guide others .

Problem solving

It is important to note at this point that critical thinking can be aimed at solving a problem but can also be used to simply answer questions or even to identify areas for improvement in certain situations. 

At Smowltech, our proctoring plans help with the creation of objective, respectful and innovative exchange and evaluation spaces.Request us a free demo in which we display all the remote supervision solutions we can offer you, as personalized and detailed reports on remote activities’ progress.

Download now!

8 interesting

about proctoring

Discover everything you need about online proctoring in this book to know how to choose the best software.

Fill out the form and download the guide now.

And subscribe to the weekly SMOWL newsletter to get exclusive offers and promotions .

You will discover all the trends in eLearning, technology, innovation, and proctoring at the hands of evaluation and talent management experts .

Discover how SMOWL works

• Register in mySmowltech indicating your LMS.
• Check your email and follow the steps to integrate the tool.
• Enjoy your free trial of 25 licenses.

Request a free demo with one of our experts

In addition to showing you how SMOWL works, we will guide and advise you at all times so that you can choose the plan that best suits your company or institution.

• Copyright © 2024 all rights reserved SMOWLTECH

Write below what you are looking for

Chapter 7: Critical Thinking and Evaluating Information

Please log in to save materials. Log in

• EPUB 3 Student View
• PDF Student View
• Thin Common Cartridge
• Thin Common Cartridge Student View
• SCORM Package
• SCORM Package Student View
• 1 - Critical Thinking and Evaluating Information
• 2 - ACTIVITY: REFLECT ON CRITICAL THINKING
• View all as one page

Critical Thinking and Evaluating Information

Critical thinking.

As a college student, you are tasked with engaging and expanding your thinking skills. One of the most important of these skills is critical thinking because it relates to nearly all tasks, situations, topics, careers, environments, challenges, and opportunities. It is a “domain-general” thinking skill, not one that is specific to a particular subject area.

What Is Critical Thinking?

Critical thinking  is clear, reasonable, reflective thinking focused on deciding what to believe or do. It means asking probing questions like “How do we know?” or “Is this true in every case or just in this instance?” It involves being skeptical and challenging assumptions rather than simply memorizing facts or blindly accepting what you hear or read.

Imagine, for example, that you’re reading a history textbook. You wonder who wrote it and why, because you detect certain biases in the writing. You find that the author has a limited scope of research focused only on a particular group within a population. In this case, your critical thinking reveals that there are “other sides to the story.”

Who are critical thinkers, and what characteristics do they have in common? Critical thinkers are usually curious and reflective people. They like to explore and probe new areas and seek knowledge, clarification, and new solutions. They ask pertinent questions, evaluate statements and arguments, and they distinguish between facts and opinion. They are also willing to examine their own beliefs, possessing a manner of humility that allows them to admit lack of knowledge or understanding when needed. They are open to changing their mind. Perhaps most of all, they actively enjoy learning, and seeking new knowledge is a lifelong pursuit. This may well be you!

No matter where you are on the road to being a critical thinker, you can always more fully develop and finely tune your skills. Doing so will help you develop more balanced arguments, express yourself clearly, read critically, and glean important information efficiently. Critical thinking skills will help you in any profession or any circumstance of life, from science to art to business to teaching. With critical thinking, you become a clearer thinker and problem solver.

Critical Thinking and Logic

Critical thinking is fundamentally a process of questioning information and data. You may question the information you read in a textbook, or you may question what a politician or a professor or a classmate says. You can also question a commonly-held belief or a new idea. With critical thinking, anything and everything is subject to question and examination for the purpose of logically constructing reasoned perspectives.

What Is Logic?

The word  logic  comes from the Ancient Greek  logike , referring to the science or art of reasoning. Using logic, a person evaluates arguments and reasoning and strives to distinguish between good and bad reasoning, or between truth and falsehood. Using logic, you can evaluate the ideas and claims of others, make good decisions, and form sound beliefs about the world. [1]

Questions of Logic in Critical Thinking

Let’s use a simple example of applying logic to a critical-thinking situation. In this hypothetical scenario, a man has a Ph.D. in political science, and he works as a professor at a local college. His wife works at the college, too. They have three young children in the local school system, and their family is well known in the community. The man is now running for political office. Are his credentials and experience sufficient for entering public office? Will he be effective in the political office? Some voters might believe that his personal life and current job, on the surface, suggest he will do well in the position, and they will vote for him. In truth, the characteristics described don’t guarantee that the man will do a good job. The information is somewhat irrelevant. What else might you want to know? How about whether the man had already held a political office and done a good job? In this case, we want to think critically about how much information is adequate in order to make a decision based on  logic  instead of  assumptions.

The following questions, presented in Figure 1, below, are ones you may apply to formulating a logical, reasoned perspective in the above scenario or any other situation:

• What’s happening?  Gather the basic information and begin to think of questions.
• Why is it important?  Ask yourself why it’s significant and whether or not you agree.
• What don’t I see?  Is there anything important missing?
• How do I know?  Ask yourself where the information came from and how it was constructed.
• Who is saying it?  What’s the position of the speaker and what is influencing them?
• What else?   What if?  What other ideas exist and are there other possibilities?

Problem-Solving with Critical Thinking

For most people, a typical day is filled with critical thinking and problem-solving challenges. In fact, critical thinking and problem-solving go hand-in-hand. They both refer to using knowledge, facts, and data to solve problems effectively. But with problem-solving, you are specifically identifying, selecting, and defending your solution. Below are some examples of using critical thinking to problem-solve:

• Your roommate was upset and said some unkind words to you, which put a crimp in the relationship. You try to see through the angry behaviors to determine how you might best support the roommate and help bring the relationship back to a comfortable spot.
• Your campus club has been languishing due to lack of participation and funds. The new club president, though, is a marketing major and has identified some strategies to interest students in joining and supporting the club. Implementation is forthcoming.
• Your final art class project challenges you to conceptualize form in new ways. On the last day of class when students present their projects, you describe the techniques you used to fulfill the assignment. You explain why and how you selected that approach.
• Your math teacher sees that the class is not quite grasping a concept. She uses clever questioning to dispel anxiety and guide you to a new understanding of the concept.
• You have a job interview for a position that you feel you are only partially qualified for, although you really want the job and you are excited about the prospects. You analyze how you will explain your skills and experiences in a way to show that you are a good match for the prospective employer.
• You are doing well in college, and most of your college and living expenses are covered. But there are some gaps between what you want and what you feel you can afford. You analyze your income, savings, and budget to better calculate what you will need to stay in college and maintain your desired level of spending.

Problem-Solving Action Checklist

Problem-solving can be an efficient and rewarding process, especially if you are organized and mindful of critical steps and strategies. Remember to assume the attributes of a good critical thinker: if you are curious, reflective, knowledge-seeking, open to change, probing, organized, and ethical, your challenge or problem will be less of a hurdle, and you’ll be in a good position to find intelligent solutions. The steps outlined in this checklist will help you adhere to these qualities in your approach to any problem:

Critical and Creative Thinking

Critical and creative thinking (described in more detail in Chapter 6: Theories of Learning) complement each other when it comes to problem-solving. The following words, by Dr. Andrew Robert Baker, are excerpted from his “Thinking Critically and Creatively” essay. Dr. Baker illuminates some of the many ways that college students will be exposed to critical and creative thinking and how it can enrich their learning experiences.

THINKING CRITICALLY AND CREATIVELY Critical thinking skills are perhaps the most fundamental skills involved in making judgments and solving problems. You use them every day, and you can continue improving them. The ability to think critically about a matter—to analyze a question, situation, or problem down to its most basic parts—is what helps us evaluate the accuracy and truthfulness of statements, claims, and information we read and hear. It is the sharp knife that, when honed, separates fact from fiction, honesty from lies, and the accurate from the misleading. We all use this skill to one degree or another almost every day. For example, we use critical thinking every day as we consider the latest consumer products and why one particular product is the best among its peers. Is it a quality product because a celebrity endorses it? Because a lot of other people may have used it? Because it is made by one company versus another? Or perhaps because it is made in one country or another? These are questions representative of critical thinking. The academic setting demands more of us in terms of critical thinking than everyday life. It demands that we evaluate information and analyze myriad issues. It is the environment where our critical thinking skills can be the difference between success and failure. In this environment we must consider information in an analytical, critical manner. We must ask questions—What is the source of this information? Is this source an expert one and what makes it so? Are there multiple perspectives to consider on an issue? Do multiple sources agree or disagree on an issue? Does quality research substantiate information or opinion? Do I have any personal biases that may affect my consideration of this information? It is only through purposeful, frequent, intentional questioning such as this that we can sharpen our critical thinking skills and improve as students, learners and researchers. While critical thinking analyzes information and roots out the true nature and facets of problems, it is creative thinking that drives progress forward when it comes to solving these problems. Exceptional creative thinkers are people that invent new solutions to existing problems that do not rely on past or current solutions. They are the ones who invent solution C when everyone else is still arguing between A and B. Creative thinking skills involve using strategies to clear the mind so that our thoughts and ideas can transcend the current limitations of a problem and allow us to see beyond barriers that prevent new solutions from being found. Brainstorming is the simplest example of intentional creative thinking that most people have tried at least once. With the quick generation of many ideas at once, we can block-out our brain’s natural tendency to limit our solution-generating abilities so we can access and combine many possible solutions/thoughts and invent new ones. It is sort of like sprinting through a race’s finish line only to find there is new track on the other side and we can keep going, if we choose. As with critical thinking, higher education both demands creative thinking from us and is the perfect place to practice and develop the skill. Everything from word problems in a math class, to opinion or persuasive speeches and papers, call upon our creative thinking skills to generate new solutions and perspectives in response to our professor’s demands. Creative thinking skills ask questions such as—What if? Why not? What else is out there? Can I combine perspectives/solutions? What is something no one else has brought-up? What is being forgotten/ignored? What about ______? It is the opening of doors and options that follows problem-identification. Consider an assignment that required you to compare two different authors on the topic of education and select and defend one as better. Now add to this scenario that your professor clearly prefers one author over the other. While critical thinking can get you as far as identifying the similarities and differences between these authors and evaluating their merits, it is creative thinking that you must use if you wish to challenge your professor’s opinion and invent new perspectives on the authors that have not previously been considered. So, what can we do to develop our critical and creative thinking skills? Although many students may dislike it, group work is an excellent way to develop our thinking skills. Many times I have heard from students their disdain for working in groups based on scheduling, varied levels of commitment to the group or project, and personality conflicts too, of course. True—it’s not always easy, but that is why it is so effective. When we work collaboratively on a project or problem we bring many brains to bear on a subject. These different brains will naturally develop varied ways of solving or explaining problems and examining information. To the observant individual we see that this places us in a constant state of back and forth critical/creative thinking modes. For example, in group work we are simultaneously analyzing information and generating solutions on our own, while challenging other’s analyses/ideas and responding to challenges to our own analyses/ideas. This is part of why students tend to avoid group work—it challenges us as thinkers and forces us to analyze others while defending ourselves, which is not something we are used to or comfortable with as most of our educational experiences involve solo work. Your professors know this—that’s why we assign it—to help you grow as students, learners, and thinkers! —Dr. Andrew Robert Baker,  Foundations of Academic Success: Words of Wisdom

Evaluating Information with Critical Thinking

Evaluating information can be one of the most complex tasks you will be faced with in college. But if you utilize the following four strategies, you will be well on your way to success:

• Read for understanding
• Examine arguments
• Clarify thinking
• Cultivate “habits of mind”

Read for Understanding

When you read, take notes or mark the text to track your thinking about what you are reading. As you make connections and ask questions in response to what you read,  you monitor your comprehension and enhance your long-term understanding of the material. You will want to mark important arguments and key facts. Indicate where you agree and disagree or have further questions. You don’t necessarily need to read every word, but make sure you understand the concepts or the intentions behind what is written. See the chapter on  Active Reading Strategies  for additional tips.

Examine Arguments

When you examine arguments or claims that an author, speaker, or other source is making, your goal is to identify and examine the hard facts. You can use the spectrum of authority strategy for this purpose. The spectrum of authority strategy assists you in identifying the “hot” end of an argument—feelings, beliefs, cultural influences, and societal influences—and the “cold” end of an argument—scientific influences. The most compelling arguments balance elements from both ends of the spectrum. The following video explains this strategy in further detail:

Clarify Thinking

When you use critical thinking to evaluate information, you need to clarify your thinking to yourself and likely to others. Doing this well is mainly a process of asking and answering probing questions, such as the logic questions discussed earlier. Design your questions to fit your needs, but be sure to cover adequate ground. What is the purpose? What question are we trying to answer? What point of view is being expressed? What assumptions are we or others making? What are the facts and data we know, and how do we know them? What are the concepts we’re working with? What are the conclusions, and do they make sense? What are the implications?

Cultivate “Habits of Mind”

“Habits of mind” are the personal commitments, values, and standards you have about the principle of good thinking. Consider your intellectual commitments, values, and standards. Do you approach problems with an open mind, a respect for truth, and an inquiring attitude? Some good habits to have when thinking critically are being receptive to having your opinions changed, having respect for others, being independent and not accepting something is true until you’ve had the time to examine the available evidence, being fair-minded, having respect for a reason, having an inquiring mind, not making assumptions, and always, especially, questioning your own conclusions—in other words, developing an intellectual work ethic. Try to work these qualities into your daily life.

In 2010, a textbook being used in fourth-grade classrooms in Virginia became big news for all the wrong reasons. The book,  Our Virginia  by Joy Masoff, had caught the attention of a parent who was helping her child do her homework, according to  an article in  The Washington Post . Carol Sheriff was a historian for the College of William and Mary and as she worked with her daughter, she began to notice some glaring historical errors, not the least of which was a passage which described how thousands of African Americans fought for the South during the Civil War.

Further investigation into the book revealed that, although the author had written textbooks on a variety of subjects, she was not a trained historian. The research she had done to write  Our Virginia,  and in particular the information she included about Black Confederate soldiers, was done through the Internet and included sources created by groups like the Sons of Confederate Veterans, an organization which promotes views of history that de-emphasize the role of slavery in the Civil War.

How did a book with errors like these come to be used as part of the curriculum and who was at fault? Was it Masoff for using untrustworthy sources for her research? Was it the editors who allowed the book to be published with these errors intact? Was it the school board for approving the book without more closely reviewing its accuracy?

There are a number of issues at play in the case of  Our Virginia , but there’s no question that evaluating sources is an important part of the research process and doesn’t just apply to Internet sources. Using inaccurate, irrelevant, or poorly researched sources can affect the quality of your own work. Being able to understand and apply the concepts that follow is crucial to becoming a more savvy user and creator of information.

When you begin evaluating sources, what should you consider? The  CRAAP test  is a series of common evaluative elements you can use to evaluate the  C urrency,  R elevance,  A uthority,  A ccuracy, and  P urpose of your sources. The CRAAP test was developed by librarians at California State University at Chico and it gives you a good, overall set of elements to look for when evaluating a resource. Let’s consider what each of these evaluative elements means. You can visit the ACC Library’s Web page for a tutorial on  Evaluating Information  using the CRAAP test.

One of the most important and interesting steps to take as you begin researching a subject is selecting the resources that will help you build your thesis and support your assertions. Certain topics require you to pay special attention to how current your resource is—because they are time sensitive, because they have evolved so much over the years, or because new research comes out on the topic so frequently. When evaluating the currency of an article, consider the following:

• When was the item written, and how frequently does the publication come out?
• Is there evidence of newly added or updated information in the item?
• If the information is dated, is it still suitable for your topic?
• How frequently does information change about your topic?

Understanding what resources are most applicable to your subject and why they are applicable can help you focus and refine your thesis. Many topics are broad and searching for information on them produces a wide range of resources. Narrowing your topic and focusing on resources specific to your needs can help reduce the piles of information and help you focus in on what is truly important to read and reference. When determining relevance consider the following:

• Does the item contain information relevant to your argument or thesis?
• Read the article’s introduction, thesis, and conclusion.
• Scan main headings and identify article keywords.
• For book resources, start with the index or table of contents—how wide a scope does the item have? Will you use part or all of this resource?
• Does the information presented support or refute your ideas?
• If the information refutes your ideas, how will this change your argument?
• Does the material provide you with current information?
• What is the material’s intended audience?

Understanding more about your information’s source helps you determine when, how, and where to use that information. Is your author an expert on the subject? Do they have some personal stake in the argument they are making? What is the author or information producer’s background? When determining the authority of your source, consider the following:

• What are the author’s credentials?
• What is the author’s level of education, experience, and/or occupation?
• What qualifies the author to write about this topic?
• What affiliations does the author have? Could these affiliations affect their position?
• What organization or body published the information? Is it authoritative? Does it have an explicit position or bias?

Determining where information comes from, if the evidence supports the information, and if the information has been reviewed or refereed can help you decide how and whether to use a source. When determining the accuracy of a source, consider the following:

• Is the source well-documented? Does it include footnotes, citations, or a bibliography?
• Is information in the source presented as fact, opinion, or propaganda? Are biases clear?
• Can you verify information from the references cited in the source?
• Is the information written clearly and free of typographical and grammatical mistakes? Does the source look to be edited before publication? A clean, well-presented paper does not always indicate accuracy, but usually at least means more eyes have been on the information.

Knowing why the information was created is a key to evaluation. Understanding the reason or purpose of the information, if the information has clear intentions, or if the information is fact, opinion, or propaganda will help you decide how and why to use information:

• Is the author’s purpose to inform, sell, persuade, or entertain?
• Does the source have an obvious bias or prejudice?
• Is the article presented from multiple points of view?
• Does the author omit important facts or data that might disprove their argument?
• Is the author’s language informal, joking, emotional, or impassioned?
• Is the information clearly supported by evidence?

When you feel overwhelmed by the information you are finding, the CRAAP test can help you determine which information is the most useful to your research topic. How you respond to what you find out using the CRAAP test will depend on your topic. Maybe you want to use two overtly biased resources to inform an overview of typical arguments in a particular field. Perhaps your topic is historical and currency means the past hundred years rather than the past one or two years. Use the CRAAP test, be knowledgeable about your topic, and you will be on your way to evaluating information efficiently and well!

Developing Yourself As a Critical Thinker

Critical thinking is a fundamental skill for college students, but it should also be a lifelong pursuit. Below are additional strategies to develop yourself as a critical thinker in college and in everyday life:

• Reflect and practice : Always reflect on what you’ve learned. Is it true all the time? How did you arrive at your conclusions?
• Use wasted time : It’s certainly important to make time for relaxing, but if you find you are indulging in too much of a good thing, think about using your time more constructively. Determine when you do your best thinking and try to learn something new during that part of the day.
• Redefine the way you see things : It can be very uninteresting to always think the same way. Challenge yourself to see familiar things in new ways. Put yourself in someone else’s shoes and consider things from a different angle or perspective.  If you’re trying to solve a problem, list all your concerns: what you need in order to solve it, who can help, what some possible barriers might be, etc. It’s often possible to reframe a problem as an opportunity. Try to find a solution where there seems to be none.
• Analyze the influences on your thinking and in your life : Why do you think or feel the way you do? Analyze your influences. Think about who in your life influences you. Do you feel or react a certain way because of social convention, or because you believe it is what is expected of you? Try to break out of any molds that may be constricting you.
• Express yourself : Critical thinking also involves being able to express yourself clearly. Most important in expressing yourself clearly is stating one point at a time. You might be inclined to argue every thought, but you might have greater impact if you focus just on your main arguments. This will help others to follow your thinking clearly. For more abstract ideas, assume that your audience may not understand. Provide examples, analogies, or metaphors where you can.
• Enhance your wellness : It’s easier to think critically when you take care of your mental and physical health. Try taking activity breaks throughout the day to reach 30 to 60 minutes of physical activity each day. Scheduling physical activity into your day can help lower stress and increase mental alertness. Also,  do your most difficult work when you have the most energy . Think about the time of day you are most effective and have the most energy. Plan to do your most difficult work during these times. And be sure to  reach out for help i f you feel you need assistance with your mental or physical health (see  Maintaining Your Mental and Physical Health  for more information).

Complete Section #2 Below: ACTIVITY: REFLECT ON CRITICAL THINKING

Key takeaways.

• Critical thinking is logical and reflective thinking focused on deciding what to believe or do.
• Critical thinking involves questioning and evaluating information.
• Critical and creative thinking both contribute to our ability to solve problems in a variety of contexts.
• Evaluating information is a complex, but essential, process. You can use the CRAAP test to help determine if sources and information are reliable.
• You can take specific actions to develop and strengthen your critical thinking skills.

ACTIVITY: REFLECT ON CRITICAL THINKING

Directions:.

• Think about someone you consider to be a critical thinker (friend, professor, historical figure, etc). What qualities does he/she have?
• Review some of the critical thinking strategies discussed on this page. Pick one strategy that makes sense to you. How can you apply this critical thinking technique to your academic work?
• Habits of mind are attitudes and beliefs that influence how you approach the world (i.e., inquiring attitude, open mind, respect for truth, etc). What is one habit of mind you would like to actively develop over the next year? How will you develop a daily practice to cultivate this habit?
• Write your responses in journal form, and submit according to your instructor’s guidelines.

• Open access
• Published: 02 September 2024

Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care

• Robin Blythe   ORCID: orcid.org/0000-0002-3643-4332 1 ,
• Sundresan Naicker   ORCID: orcid.org/0000-0002-2392-4981 1 ,
• Nicole White   ORCID: orcid.org/0000-0002-9292-0773 1 ,
• Raelene Donovan   ORCID: orcid.org/0000-0003-0737-7719 2 ,
• Ian A. Scott   ORCID: orcid.org/0000-0002-7596-0837 3 , 4 ,
• Andrew McKelliget 2 &
• Steven M McPhail   ORCID: orcid.org/0000-0002-1463-662X 1 , 4  

BMC Medical Informatics and Decision Making volume  24 , Article number:  241 ( 2024 ) Cite this article

Metrics details

Successful deployment of clinical prediction models for clinical deterioration relates not only to predictive performance but to integration into the decision making process. Models may demonstrate good discrimination and calibration, but fail to match the needs of practising acute care clinicians who receive, interpret, and act upon model outputs or alerts. We sought to understand how prediction models for clinical deterioration, also known as early warning scores (EWS), influence the decision-making of clinicians who regularly use them and elicit their perspectives on model design to guide future deterioration model development and implementation.

Nurses and doctors who regularly receive or respond to EWS alerts in two digital metropolitan hospitals were interviewed for up to one hour between February 2022 and March 2023 using semi-structured formats. We grouped interview data into sub-themes and then into general themes using reflexive thematic analysis. Themes were then mapped to a model of clinical decision making using deductive framework mapping to develop a set of practical recommendations for future deterioration model development and deployment.

Fifteen nurses ( n  = 8) and doctors ( n  = 7) were interviewed for a mean duration of 42 min. Participants emphasised the importance of using predictive tools for supporting rather than supplanting critical thinking, avoiding over-protocolising care, incorporating important contextual information and focusing on how clinicians generate, test, and select diagnostic hypotheses when managing deteriorating patients. These themes were incorporated into a conceptual model which informed recommendations that clinical deterioration prediction models demonstrate transparency and interactivity, generate outputs tailored to the tasks and responsibilities of end-users, avoid priming clinicians with potential diagnoses before patients were physically assessed, and support the process of deciding upon subsequent management.

Conclusions

Prediction models for deteriorating inpatients may be more impactful if they are designed in accordance with the decision-making processes of acute care clinicians. Models should produce actionable outputs that assist with, rather than supplant, critical thinking.

• This article explored decision-making processes of clinicians using a clinical prediction model for deteriorating patients, also known as an early warning score.

• Our study identified that the clinical utility of deterioration models may lie in their assistance in generating, evaluating, and selecting diagnostic hypotheses, an important part of clinical decision making that is underrepresented in the prediction modelling literature.

• Nurses in particular stressed the need for models that encourage critical thinking and further investigation rather than prescribe strict care protocols.

Peer Review reports

The number of ‘clinical prediction model’ articles published on PubMed has grown rapidly over the past two decades, from 1,918 articles identified with these search terms published in 2002 to 26,326 published in 2022. A clinical prediction model is defined as any multivariable model that provides patient-level estimates of the probability or risk of a disease, condition or future event [ 1 , 2 , 3 ].

Recent systematic and scoping reviews report a lack of evidence that clinical decision support systems based on prediction models are associated with improved patient outcomes once implemented in acute care [ 4 , 5 , 6 , 7 ]. One potential reason may be that some models are not superior to clinical judgment in reducing missed diagnoses or correctly classifying non-diseased patients [ 8 ]. While improving predictive accuracy is important, this appears insufficient for improving patient outcomes, suggesting that more attention should be paid to the process and justification of how prediction models are designed and deployed [ 9 , 10 ].

If model predictions are to influence clinical decision-making, they must not only demonstrate acceptable accuracy, but also be implemented and adopted at scale in clinical settings. This requires consideration of how they are integrated into clinical workflows, how they generate value for users, and how clinicians perceive and respond to their outputs of predicted risks [ 11 , 12 ]. These concepts are tenets of user-centred design, which focuses on building systems based on the needs and responsibilities of those who will use them. User-centred decision support tools can be designed in a variety of ways, but may benefit from understanding the characteristics of the users and the local environment in which tools are implemented, [ 13 ] the nature of the tasks end-users are expected to perform, [ 14 ] and the interface between the user and the tools [ 15 ].

Prediction models for clinical deterioration

A common task for prediction models integrated into clinical decision support systems is in predicting or recognising clinical deterioration, also known as early warning scores. Clinical deterioration is defined as the transition of a patient from their current health state to a worse one that puts them at greater risk of adverse events and death [ 16 ]. Early warning scores were initially designed to get the attention of skilled clinicians when patients began to deteriorate, but have since morphed into complex multivariable prediction models [ 17 ]. As with many other clinical prediction models, early warning scores often fail to demonstrate better patient outcomes once deployed [ 4 , 18 ]. The clinical utility of early warning scores likely rests on two key contextual elements: the presence of uncertainty, both in terms of diagnosis and prognosis, and the potential for undesirable patient outcomes if an appropriate care pathway is delayed or an inappropriate one is chosen [ 19 ].

The overarching goal of this qualitative study was to determine how prediction models for clinical deterioration, or early warning scores, could be better tailored to the needs of end-users to improve inpatient care. This study had three aims. First, to understand the experiences and perspectives of nurses and doctors who use early warning scores. Second, to identify the tasks these clinicians performed when managing deteriorating patients, the decision-making processes that guided these tasks, and how these could be conceptualised schematically. Finally, to address these tasks and needs with actionable, practical recommendations for enhancing future deterioration prediction model development and deployment.

To achieve our study aims, we conducted semi-structured interviews of nurses and doctors at two large, digitally mature hospitals. We first asked clinicians to describe their backgrounds, perspectives, and experience with early warning scores to give context to our analysis. We then examined the tasks and responsibilities of participants and the decision-making processes that guided these tasks using reflexive thematic analysis, an inductive method that facilitated the identification of general themes. We then identified a conceptual decision-making framework from the literature to which we mapped these themes to understand how they may lead to better decision support tools. Finally, we used this framework to formulate recommendations for deterioration prediction model design and deployment. These steps are presented graphically in a flow diagram (Fig.  1 ).

Schema of study goal, aims and methods

The study was conducted at one large tertiary and one medium-sized metropolitan hospital in Brisbane, Australia. The large hospital contained over 1,000 beds, handling over 116,000 admissions and approximately 150,000 deterioration alerts per year in 2019. Over the same period, the medium hospital contained 175 beds, handling over 31,000 admissions and approximately 42,000 deterioration alerts per year. These facilities had a high level of digital maturity, including fully integrated electronic medical records.

Clinical prediction model for deteriorating patients

The deterioration monitoring system used at both hospitals was the Queensland Adult Deterioration Detection System (Q-ADDS) [ 20 , 21 ]. Q-ADDS uses an underlying prediction model to convert patient-level vital signs from a single time of observation into an ordinal risk score describing an adult patient’s risk of acute deterioration. Vital signs collected are respiratory rate (breaths/minute), oxygen flow rate (L/minute), arterial oxygen saturation (percent), blood pressure (mmHg), heart rate (beats/minute), temperature (degrees Celsius), level of consciousness (Alert-Voice-Pain-Unresponsive) and increased or new onset agitation. Increased pain and urine output are collected but not used for score calculation [ 21 ]. The Q-ADDS tool is included in the supplementary material.

Vital signs are entered into the patient’s electronic medical record, either imported from the vital signs monitoring device at the patient’s bedside or from manual entry by nurses. Calculations are made automatically within Q-ADDS to generate an ordinal risk score per patient observation. Scores can be elevated to levels requiring a tiered escalation response if a single vital sign is greatly deranged, or if several observations are deranged by varying degrees. Scores range from 0 to 8+, with automated alerts and escalation protocols ranging from more frequent observations for lower scores to immediate activation of the medical emergency team (MET) at higher scores.

The escalation process for Q-ADDS is highly structured, mandated and well documented [ 21 ]. Briefly, when a patient’s vital signs meet a required alert threshold, the patient’s nurse is required to physically assess the patient and, depending on the level of severity predicted by Q-ADDS, notify the patient’s doctor (escalation). The doctor is then required to be notified of the patient’s Q-ADDS score, potentially review the patient, and discuss any potential changes to care with the nurse. Both nurses and doctors can escalate straight to MET calls or an emergency ‘code blue’ call (requiring cardiopulmonary resuscitation or assisted ventilation) at any time if necessary.

Participant recruitment

Participant recruitment began in February 2022 and concluded in March 2023, disrupted by the COVID-19 pandemic. Eligibility criteria were nurses or doctors at each hospital with direct patient contact who either receive or respond, respectively, to Q-ADDS alerts. An anticipated target sample size of 15 participants was established prior to recruitment, based on expected constraints in recruitment due to clinician workloads and the expected length of interviews relative to their scope, as guided by prior research [ 22 ]. As the analysis plan involved coding interviews iteratively as they were conducted, the main justification for ceasing recruitment was when no new themes relating to the study objectives were generated during successive interviews as the target sample size was approached [ 23 ].

Study information was broadly distributed via email to nurses and doctors in patient-facing roles across hospitals. Nurse unit managers were followed up during regular nursing committee meetings to participate or assist with recruitment within their assigned wards. Doctors were followed up by face-to-face rounding. Snowball sampling, in which participants were encouraged to refer their colleagues for study participation, was employed whenever possible. In all cases, study authors explained study goals and distributed participant consent forms prior to interview scheduling with the explicit proviso that participation was completely voluntary and anonymous to all but two study authors (RB and SN).

Interview process

We used a reflexive framework method to develop an open-ended interview template [ 24 ] that aligned with our study aims. Interview questions were informed by the non-adoption, abandonment, scale-up, spread and sustainability (NASSS) framework [ 25 ]. The NASSS framework relates the end-user perceptions of the technology being evaluated to its value proposition for the clinical situation to which it is being applied. We selected a reflexive method based on the NASSS for our study as we wanted to allow end-users to speak freely about the barriers they faced when using prediction models for clinical deterioration, but did not limit participants to discussing only topics that could fit within the NASSS framework.

Participants were first asked about their background and clinical expertise. They were then invited to share their experiences and perspectives with using early warning scores to manage deteriorating patients. This was used as a segue for participants to describe the primary tasks required of them when evaluating and treating a deteriorating patient. Participants were encouraged to talk through their decision-making process when fulfilling these tasks, and to identify any barriers or obstacles to achieving those tasks that were related to prediction models for deteriorating patients. Participants were specifically encouraged to identify any sources of information that were useful for managing deteriorating patients, including prediction models for other, related disease groups like sepsis, and to think of any barriers or facilitators for making that information more accessible. Finally, participants were invited to suggest ways to improve early warning scores, and how those changes may lead to benefits for patients and clinicians.

As we employed a reflexive methodology to allow clinicians to speak freely about their perspectives and opinions, answers to interview questions were optional and open-ended, allowing participants to discuss relevant tangents. Separate interview guides were developed for nurses and doctors as the responsibilities and information needs of these two disciplines in managing deteriorating patients often differ. Nurses are generally charged with receiving and passing on deterioration alerts, while doctors are generally charged with responding to alerts and making any required changes to patient care plans [ 4 ]. Interview guides are contained in the supplement.

Due to clinician workloads, member checking, a form of post-interview validation in which participants retrospectively confirm their interview answers, was not used. To ensure participants perceived the interviewers as being impartial, two study authors not employed by the hospital network and not involved in direct patient care (RB and SN) were solely responsible for conducting interviews and interrogating interview transcripts. Interviews were recorded and transcribed verbatim, then re-checked for accuracy.

Inductive thematic analysis

Transcripts were analysed using a reflexive thematic methodology informed by Braun and Clarke [ 26 ]. This method was selected because it facilitated exploring the research objectives rather than being restricted to the domains of a specific technology adoption framework, which may limit generalisability [ 27 ]. Interviews were analysed over five steps to identify emergent themes.

Each interview was broken down into segments by RB and SN, where segments corresponded to a distinct opinion.

Whenever appropriate, representative quotes for each distinct concept were extracted.

Segments were grouped into sub-themes.

Sub-themes were grouped into higher-order themes, or general concepts.

Steps 1 through 4 were iteratively repeated by RB and supervised by SN.

As reflexive methods incorporate the experiences and expertise of the analysts, our goal was to extract any sub-themes relevant to the study aims and able to be analysed in the context of early warning scores, prediction models, or decision support tools for clinical deterioration. The concepts explored during this process were not exhaustive, but repeated analysis and re-analysis of participant transcripts helped to ensure all themes could be interpreted in the context of our three study aims: background and perspectives, tasks and decision-making, and recommendations for future practice.

Deductive mapping to a clinical decision-making framework

Once the emergent themes from the inductive analysis were defined, we conducted a brief scan of PubMed for English-language studies that investigated how the design of clinical decision support systems relate to clinical decision-making frameworks. The purpose of this exercise was to identify a framework against which we could map the previously elicited contexts, tasks, and decision-making of end-users in developing a decision-making model that could then be used to support the third aim of formulating recommendations to enhance prediction model development and deployment.

RB and SN then mapped higher-order themes from the inductive analysis to the decision-making model based on whether there was a clear relationship between each theme and a node in the model (see Results).

Recommendations for improving prediction model design were derived by reformatting the inductive themes based on the stated preferences of the participants. These recommendations were then assessed by the remaining authors and the process repeated iteratively until authors were confident that all recommendations were concordant with the decision-making model.

Participant characteristics

Our sample included 8 nurses and 7 doctors of varying levels of expertise and clinical specialties; further information is contained in the supplement. Compared to doctors, nurse participants were generally more experienced, often participating in training or mentoring less experienced staff. Clinical specialities of nurses were diverse, including orthopaedics, cancer services, medical assessment and planning unit, general medicine, and pain management services. Doctor participants ranged from interns with less than a year of clinical experience up to consultant level, including three doctors doing training rotations and two surgical registrars. Clinical specialties of doctors included geriatric medicine, colorectal surgery, and medical education.

Interviews and thematic analysis

Eleven interviews were conducted jointly by RB and SN, one conducted by RB, and three by SN. Interviews were scheduled for up to one hour, with a mean duration of 42 min. Six higher-order themes were identified. These were: added value of more information; communication of model outputs; validation of clinical intuition; capability for objective measurement; over-protocolisation of care; and model transparency and interactivity (Table  1 ). Some aspects of care, including the need for critical thinking and the informational value of discerning trends in patient observations, were discussed in several contexts, making them relevant to more than one higher-order theme.

Added value of other information

Clinicians identified that additional data or variables important for decision making were often omitted from the Q-ADDS digital interface. Such variables included current medical conditions, prescribed medications and prior observations, which were important for interpreting current patient data in the context of their baseline observations under normal circumstances (e.g., habitually low arterial oxygen saturation due to chronic obstructive pulmonary disease) or in response to an acute stimulus (e.g., expected hypotension for next 4 to 8 h while treatment for septic shock is underway).

“The trend is the biggest thing [when] looking at the data , because sometimes people’s observations are deranged forever and it’s not abnormal for them to be tachycardic , whereas for someone else , if it’s new and acute , then that’s a worry.” – Registrar.

Participants frequently emphasised the critical importance of looking at patients holistically, or that patients were more than the sum of the variables used to predict risk. Senior nurses stressed that prediction models were only one part of patient evaluation, and clinicians should be encouraged to incorporate both model outputs and their own knowledge and experiences in decision making rather than trust models implicitly. Doctors also emphasised this holistic approach, adding that they placed more importance on hearing a nurse was concerned for the patient than seeing the model output. Critical thinking about future management was frequently raised in this context, with both nurses and doctors insisting that model predictions and the information required for contextualising risk scores should be communicated together when escalating the patient’s care to more senior clinicians.

Model outputs

Model outputs were discussed in two contexts. First, doctors perceived that ordinal risk scores generated by Q-ADDS felt arbitrary compared to receiving probabilities of a future event, for example cardiorespiratory decompensation, that required a response such as resuscitation or high-level treatment. However, nurses did not wholly embrace probabilities as outputs, instead suggesting that recommendations for how they should respond to different Q-ADDS scores were more important. This difference may reflect the different roles of alert receivers (nurses) and alert responders (doctors).

“[It’s helpful] if you use probabilities… If your patient has a sedation score of 2 and a respiratory rate of 10 , [giving them] a probability of respiratory depression would be helpful. However , I don’t find many clinicians , and certainly beginning practitioners , think in terms of probabilities.” – Clinical nurse consultant.

Second, there was frequent mention of alert fatigue in the context of model outputs. One doctor and two nurses felt there was insufficient leeway for nurses to exercise discretion in responding to risk scores, leading to many unnecessary alert-initiated actions. More nuance in the way Q-ADDS outputs were delivered to clinicians with different roles was deemed important to avoid model alerts being perceived as repetitive and unwarranted. However, three other doctors warned against altering MET call criteria in response to repetitive and seemingly unchanging risk scores and that at-risk patients should, as a standard of care, remain under frequent observation. Frustrations centred more often around rigidly tying repetitive Q-ADDS outputs to certain mandated actions, leading to multiple clinical reviews in a row for a patient whose trajectory was predictable, for example a patient with stable heart failure having a constantly low blood pressure. This led to duplication of nursing effort (e.g., repeatedly checking the blood pressure) and the perception that prediction models were overly sensitive.

“It takes away a lot of nurses’ critical judgement. If someone’s baseline systolic [blood pressure] is 95 [mmHg] , they’re asymptomatic and I would never hear about it previously. We’re all aware that this is where they sit and that’s fine. Now they are required to notify me in the middle of the night , “Just so you know , they’ve dropped to 89 [below an alert threshold of 90mmHg].“” – Junior doctor.

Validation of clinical intuition

Clinicians identified the ability of prediction models to validate their clinical intuition as both a benefit and a hindrance, depending on how outputs were interpreted and acted upon. Junior clinicians appreciated early warning scores giving them more support to escalate care to senior clinicians, as a conversation starter or framing a request for discussion. Clinicians described how assessing the patient holistically first, then obtaining model outputs to add context and validate their diagnostic hypotheses, was very useful in deciding what care should be initiated and when.

“You kind of rule [hypotheses] out… you go to the worst extreme: is it something you need to really be concerned about , especially if their [score] is quite high? You’re thinking of common complications like blood clots , so that presents as tachycardic… I’m thinking of a PE [pulmonary embolism] , then you do the nursing interventions.” – Clinical nurse manager.

While deterioration alerts were often seen as triggers to think about potential causes for deterioration, participants noted that decision making could be compromised if clinicians were primed by model outputs to think of different diagnoses before they had fully assessed the patient at the bedside. Clinicians described the dangers of tunnel vision or, before considering all available clinical information, investigating favoured diagnoses to the exclusion of more likely causes.

“[Diagnosis-specific warnings are] great , [but] that’s one of those things that can lead to a bit of confirmation bias… It’s a good trigger to articulate , “I need to look for sources of infection when I go to escalate"… but then , people can get a little bit sidetracked with that and ignore something more blatant in front of them. I’ve seen people go down this rabbit warren of being obsessed with the “fact” that it was sepsis , but it was something very , very unrelated.” – Nurse educator.

Objective measurement

Clinicians perceived that prediction models were useful as more objective measures of patients’ clinical status that could ameliorate clinical uncertainty or mitigate cognitive biases. In contrast to the risk of confirmation bias arising from front-loading model outputs suggesting specific diagnoses, prediction models could offer a second opinion that could help clinicians recognise opposing signals in noisy data that, in particular, assisted in considering serious diagnoses that shouldn’t be missed (e.g., sepsis), or more frequent and easily treated diagnoses (e.g., dehydration). Prediction models were also useful when they disclosed several small, early changes in patient status that provided an opportunity for early intervention.

“Maybe [the patient has] a low grade fever , they’re a bit tachycardic. Maybe [sepsis] isn’t completely out of the blue for this person. If there was some sort of tool , that said there’s a reasonable chance that they could have sepsis here , I would use that to justify the option of going for blood cultures and maybe a full septic screen. If [I’m indecisive] , that sort of information could certainly push me in that direction.” – Junior doctor.

Clinicians frequently mentioned that prediction models would have been more useful when first starting clinical practice, but become less useful with experience. However, clinicians noted that at any experience level, risk scoring was considered most useful as a triage/prioritisation tool, helping decide which patients to see first, or which clinical concerns to address first.

“[Doctors] can easily triage a patient who’s scoring 4 to 5 versus 1 to 3. If they’re swamped , they can change the escalation process , or triage appropriately with better communication.” – Clinical nurse manager.

Clinicians also stressed that predictions were not necessarily accurate because measurement error or random variation, especially one-off outlier values for certain variables, was a significant contributor to false alerts and inappropriate responses. For example, a single unusually high respiratory rate generated an unusually high risk score, prompting an unnecessary alert.

Over-protocolisation of care

The sentiment most commonly expressed by all experienced nursing participants and some doctors was that nurses were increasingly being trained to solely react to model outputs with fixed response protocols, rather than think critically about what is happening to patients and why. It was perceived that prediction models may actually reduce the capacity for clinicians to process and internalise important information. For example, several nurses observed their staff failing to act on their own clinical suspicions that patients were deteriorating because the risk score had not exceeded a response threshold.

“We’ve had patients on the ward that have had quite a high tachycardia , but it’s not triggering because it’s below the threshold to trigger… [I often need to make my staff] make the clinical decision that they can call the MET anyway , because they have clinical concern with the patient.” – Clinical nurse consultant.

A source of great frustration for many nurses was the lack of critical thinking by their colleagues of possible causes when assessing deteriorating patients. They wanted their staff to investigate whether early warning score outputs or other changes in patient status were caused by simple, easily fixable issues such as fitting the oxygen mask properly and helping the patient sit up to breathe more easily, or whether they indicated more serious underlying pathophysiology. Nurses repeatedly referenced the need for clinicians to always be asking why something was happening, not simply reacting to what was happening.

“[Models should also be] trying to get back to critical thinking. What I’m seeing doesn’t add up with the monitor , so I should investigate further than just simply calling the code.” – Clinical nurse educator.

Model transparency and interactivity

Clinicians frequently requested more transparent and interactive prediction models. These included a desire to receive more training in how prediction models worked and how risk estimates were generated mathematically, and being able to visualise important predictors of deterioration and the absolute magnitude of their effects (effect sizes) in intuitive ways. For example, despite receiving training in Q-ADDS, nurses expressed frustrations that nobody at the hospital seemed to understand how it worked in generating risk scores. Doctors were interested in being able to visualise the relative size and direction of effect of different model variables, potentially using colour-coding, combined with other contextual patient data like current vital sign trends and medications, and presented on one single screen.

The ability to modify threshold values for model variables and see how this impacted risk scores, and what this may then mean for altering MET calling criteria, was also discussed. For example, in an older patient with an acute ischaemic stroke, a persistently high, asymptomatic blood pressure value is an expected bodily response to this acute insult over the first 24–48 h. In the absence of any change to alert criteria, recurrent alerts would be triggered which may encourage overtreatment and precipitous lowering of the blood pressure with potential to cause harm. Altering the criteria to an acceptable or “normal” value for this clinical scenario (i.e. a higher than normal blood pressure) may generate a lower, more patient-centred risk estimate and less propensity to overtreat. This ability to tinker with the model may also enhance understanding of how it works.

“I wish I could alter criteria and see what the score is after that , with another set of observations. A lot of the time… I wonder what they’re sitting at , now that I’ve [altered] the bit that I’m not concerned about… It would be quite helpful to refresh it and have their score refreshed as the new score.” – Junior doctor.

Derivation of the decision-making model

Guided by the responses of our participants regarding their decision-making processes, our literature search identified a narrative review by Banning (2008) that reported previous work by O’Neill et al. (2005) [ 28 , 29 ]. While these studies referred to models of nurse decision-making, we selected a model (Fig.  2 ) that also appropriately described the responses of doctors in our participant group and matched the context of using clinical decision support systems to support clinical judgement. As an example, when clinicians referenced needing to look for certain data points to give context to a patient assessment, this was mapped to nodes relating to “Current patient data,” “Changes to patient status/data,” and “Hypothesis-driven assessment.”

Decision-making model(Adapted from Neill’s clinical decision making framework [2005] and modified by Banning [2006]) with sequential decision nodes

Mapping of themes to decision-making model

The themes from Table  1 were mapped to the nodes in the decision-making model based on close alignment with participant responses (see Fig.  3 ). This mapping is further explained below, where the nodes in the model are described in parentheses.

Value of additional information for decision-making : participants stressed the importance of understanding not only the data going into the prediction model, but also how that data changed over time as trends, and the data that were not included in the model. (Current patient data, changes to patient status/data)

Format, frequency, and relevance of outputs : participants suggested a change in patient data should not always lead to an alert. Doctors, but not necessarily nurses, proposed outputs displayed as probabilities rather than scores, tying model predictions to potential diagnoses or prognoses. (Changes to patient status/data, hypothesis generation)

Using models to validate but not supersede clinical intuition : Depending on the exact timing of model outputs within the pathway of patient assessment, participants found predictions could either augment or hinder the hypothesis generation process. (Hypothesis generation)

Measuring risks objectively : Risk scores can assist with triaging or prioritising patients by urgency or prognostic risk, thereby potentially leading to early intervention to identify and/or prevent adverse events. (Clinician concerns, hypothesis generation)

Supporting critical thinking and reducing over-protocolised care : by acting as triggers for further assessment, participants suggested prediction models can support or discount diagnostic hypotheses, lead to root-cause identification, and facilitate interim cares, for example by ensuring good fit of nasal prongs. (Provision of interim care, hypothesis generation, hypothesis-driven assessment)

Model transparency and interactivity : understanding how prediction models worked, being able to modify or add necessary context to model predictions, and understanding the relative contribution of different predictors could better assist the generation and selection of different hypotheses that may explain a given risk score. (Hypothesis generation, recognition of clinical pattern and hypothesis selection)

Mapping of the perceived relationships between higher-order themes and nodes in the decision-making model shown in Fig.  2

Recommendations for improving the design of prediction models

Based on the mapping of themes to the decision-making model, we formulated four recommendations for enhancing the development and deployment of prediction models for clinical deterioration.

Improve accessibility and transparency of data included in the model. Provide an interface that allows end-users to see what predictor variables are included in the model, their relative contributions to model outputs, and facilitate easy access to data not included in the model but still relevant for model-informed decisions, e.g., trends of predictor variables over time.

Present model outputs that are relevant to the end-user receiving those outputs, their responsibilities, and the tasks they may be obliged to perform, while preserving the ability of clinicians to apply their own discretionary judgement.

In situations associated with diagnostic uncertainty, avoid tunnel vision from priming clinicians with possible diagnostic explanations based on model outputs, prior to more detailed clinical assessment of the patient.

Support critical thinking whereby clinicians can apply a more holistic view of the patient’s condition, take all relevant contextual factors into account, and be more thoughtful in generating and selecting causal hypotheses.

This qualitative study involving front-line acute care clinicians who respond to early warning score alerts has generated several insights into how clinicians perceive the use of prediction models for clinical deterioration. Clinicians preferred models that facilitated critical thinking, allowed an understanding of the impact of variables included and excluded from the model, provided model outputs specific to the tasks and responsibilities of different disciplines of clinicians, and supported decision-making processes in terms of hypotheses and choice of management, rather than simply responding to alerts in a pre-specified, mandated manner. In particular, preventing prediction models from supplanting critical thinking was repeatedly emphasised.

Reduced staffing ratios, less time spent with patients, greater reliance on more junior workforce, and increasing dependence on automated activation of protocolised management are all pressures that could lead to a decline in clinical reasoning skills. This problem could be exacerbated by adding yet more predictive algorithms and accompanying protocols for other clinical scenarios, which may intensify alert fatigue and disrupt essential clinical care. However, extrapolating our results to areas other than clinical deterioration should be done with caution. An opposing view may be that using prediction models to reduce the burden of routine surveillance may allow redirection of critical thinking skills towards more useful tasks, a question that has not been explored in depth in the clinical informatics literature.

Clinicians expressed interest in models capable of providing causal insights into clinical deterioration. This is neither a function nor capability of most risk prediction models, requiring different assumptions and theoretical frameworks [ 30 ]. Despite this limitation, risk nomograms, visualisations of changes in risk with changes in predictor variables, and other interactive tools for estimating risk may be useful adjuncts for clinical decision-making due to the ease with which input values can be manipulated.

Contributions to the literature

Our research supports and extends the literature on the acceptability of risk prediction models within clinical decision support systems. Common themes in the literature supporting good practices in clinical informatics and which are also reflected in our study include: alert fatigue; the delivery of more relevant contextual information; [ 31 ] the value of patient histories; [ 32 , 33 ] ranking relevant information by clinical importance, including colour-coding; [ 34 , 35 ] not using computerised tools to replace clinical judgement; [ 32 , 36 , 37 ] and understanding the analytic methods underpinning the tool [ 38 ]. One other study has investigated the perspectives of clinicians of relatively simple, rules-based prediction models similar to Q-ADDS. Kappen et al [ 12 ] conducted an impact study of a prediction model for postoperative nausea and vomiting and also found that clinicians frequently made decisions in an intuitive manner that incorporated information both included and absent from prediction models. However, the authors recommended a more directive than assistive approach to model-based recommendations, possibly due to a greater focus on timely prescribing of effective prophylaxis or treatment.

The unique contribution of our study is a better understanding of how clinicians may use prediction models to generate and validate diagnostic hypotheses. The central role of critical thinking and back-and-forth interactions between clinician and model in our results provide a basis for future research using more direct investigative approaches like cognitive task analysis [ 39 ]. Our study has yielded a set of cognitive insights into decision making that can be applied in tandem with statistical best practice in designing, validating and implementing prediction models. [ 19 , 40 , 41 ].

Relevance to machine learning and artificial intelligence prediction models for deterioration

Our results may generalise to prediction models based on machine learning (ML) and artificial intelligence (AI), according to results of several recent studies. Tonekaboni et al [ 42 ] investigated clinician preferences for ML models in the intensive care unit and emergency department using hypothetical scenarios. Several themes appear both in our results and theirs: a need to understand the impact of both included and excluded predictors on model performance; the role of uncertain or noisy data in prediction accuracy; and the influence of trends or patient trajectories in decision making. Their recommendations for more transparent models and the delivery of model outputs designed for the task at hand align closely with ours. The authors’ focus on clinicians’ trust in the model was not echoed by our participants.

Eini-Porat et al [ 43 ] conducted a comprehensive case study of ML models in both adult and paediatric critical care. Their results present several findings supported by our participants despite differences in clinical environments: the value of trends and smaller changes in several vital signs that could cumulatively signal future deterioration; the utility of triage and prioritisation in time-poor settings; and the use of models as triggers for investigating the cause of deterioration.

As ML/AI models proliferate in the clinical deterioration prediction space, [ 44 ] it is important to deeply understand the factors that may influence clinician acceptance of more complex approaches. As a general principle, these methods often strive to input as many variables or transformations of those variables as possible into the model development process to improve predictive accuracy, incorporating dynamic updating to refine model performance. While this functionality may be powerful, highly complex models are not easily explainable, require careful consideration of generalisability, and can prevent clinicians from knowing when a model is producing inaccurate predictions, with potential for patient harm when critical healthcare decisions are being made [ 45 , 46 , 47 ]. Given that our clinicians emphasised the need to understand the model, know which variables are included and excluded, and correctly interpret the format of the output, ML/AI models in the future will need to be transparent in their development and their outputs easily interpretable.

Limitations

The primary limitations of our study were that our sample was drawn from two hospitals with high levels of digital maturity in a metropolitan region of a developed country, with a context specific to clinical deterioration. Our sample of 15 participants may be considered small but is similar to that of other studies with a narrow focus on clinical perspectives [ 42 , 43 ]. All these factors can limit generalisability to other settings or to other prediction models. As described in the methods, we used open-ended interview templates and generated our inductive themes reflexively, which is vulnerable to different types of biases compared to more structured preference elicitation methods with rigidly defined analysis plans. Member checking may have mitigated this bias, but was not possible due to the time required from busy clinical staff.

Our study does not directly deal with methodological issues in prediction model development, [ 41 , 48 ] nor does it provide explicit guidance on how model predictions should be used in clinical practice. Our findings should also not be considered an exhaustive list of concerns clinicians have with prediction models for clinical deterioration, nor may they necessarily apply to highly specialised clinical areas, such as critical care. Our choice of decision making framework was selected because it demonstrated a clear, intuitive causal pathway for model developers to support the clinical decision-making process. However, other, equally valid frameworks may have led to different conclusions, and we encourage more research in this area.

This study elicited clinician perspectives of models designed to predict and manage impending clinical deterioration. Applying these perspectives to a decision-making model, we formulated four recommendations for the design of future prediction models for deteriorating patients: improved transparency and interactivity, tailoring models to the tasks and responsibilities of different end-users, avoiding priming clinicians with diagnostic predictions prior to in-depth clinical review, and finally, facilitating the diagnostic hypothesis generation and assessment process.

Availability of data and materials

Due to privacy concerns and the potential identifiability of participants, interview transcripts are not available. However, interview guides are available in the supplement.

Jenkins DA, Martin GP, Sperrin M, Riley RD, Debray TPA, Collins GS, Peek N. Continual updating and monitoring of clinical prediction models: time for dynamic prediction systems? Diagn Prognostic Res. 2021;5(1):1.

Article   Google Scholar  

Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.

Article   PubMed   Google Scholar  

Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73.

Blythe R, Parsons R, White NM, Cook D, McPhail SM. A scoping review of real-time automated clinical deterioration alerts and evidence of impacts on hospitalised patient outcomes. BMJ Qual Saf. 2022;31(10):725–34.

Fahey M, Crayton E, Wolfe C, Douiri A. Clinical prediction models for mortality and functional outcome following ischemic stroke: a systematic review and meta-analysis. PLoS ONE. 2018;13(1):e0185402.

Article   PubMed   PubMed Central   Google Scholar  

Fleuren LM, Klausch TLT, Zwager CL, Schoonmade LJ, Guo T, Roggeveen LF, et al. Machine learning for the prediction of sepsis: a systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020;46(3):383–400.

White NM, Carter HE, Kularatna S, Borg DN, Brain DC, Tariq A, et al. Evaluating the costs and consequences of computerized clinical decision support systems in hospitals: a scoping review and recommendations for future practice. J Am Med Inform Assoc. 2023;30(6):1205–18.

Sanders S, Doust J, Glasziou P. A systematic review of studies comparing diagnostic clinical prediction rules with clinical judgment. PLoS ONE. 2015;10(6):e0128233.

Abell B, Naicker S, Rodwell D, Donovan T, Tariq A, Baysari M, et al. Identifying barriers and facilitators to successful implementation of computerized clinical decision support systems in hospitals: a NASSS framework-informed scoping review. Implement Sci. 2023;18(1):32.

van der Vegt AH, Campbell V, Mitchell I, Malycha J, Simpson J, Flenady T, et al. Systematic review and longitudinal analysis of implementing Artificial Intelligence to predict clinical deterioration in adult hospitals: what is known and what remains uncertain. J Am Med Inf Assoc. 2024;31(2):509–24.

Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Future Healthc J. 2019;6(2):94–8.

Kappen TH, van Loon K, Kappen MA, van Wolfswinkel L, Vergouwe Y, van Klei WA, et al. Barriers and facilitators perceived by physicians when using prediction models in practice. J Clin Epidemiol. 2016;70:136–45.

Witteman HO, Dansokho SC, Colquhoun H, Coulter A, Dugas M, Fagerlin A, Giguere AM, Glouberman S, Haslett L, Hoffman A, Ivers N. User-centered design and the development of patient decision aids: protocol for a systematic review. Systematic reviews. 2015;4:1−8.

Zhang J, Norman DA. Representations in distributed cognitive tasks. Cogn Sci. 1994;18(1):87–122.

Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inf. 2005;38(1):75–87.

Jones D, Mitchell I, Hillman K, Story D. Defining clinical deterioration. Resuscitation. 2013;84(8):1029–34.

Morgan RJ, Wright MM. In defence of early warning scores. Br J Anaesth. 2007;99(5):747–8.

Article   CAS   PubMed   Google Scholar  

Smith ME, Chiovaro JC, O’Neil M, Kansagara D, Quinones AR, Freeman M, et al. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Annals Am Thorac Soc. 2014;11(9):1454–65.

Baker T, Gerdin M. The clinical usefulness of prognostic prediction models in critical illness. Eur J Intern Med. 2017;45:37–40.

Campbell V, Conway R, Carey K, Tran K, Visser A, Gifford S, et al. Predicting clinical deterioration with Q-ADDS compared to NEWS, between the flags, and eCART track and trigger tools. Resuscitation. 2020;153:28–34.

The Australian Commission on Safety and Quality in Health is the publisher, and the publisher location is Sydney, Australia. https://www.safetyandquality.gov.au/sites/default/files/migrated/35981-ChartDevelopment.pdf .

Vasileiou K, Barnett J, Thorpe S, Young T. Characterising and justifying sample size sufficiency in interview-based studies: systematic analysis of qualitative health research over a 15-year period. BMC Med Res Methodol. 2018;18(1):148.

Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are Enough? Qual Health Res. 2017;27(4):591–608.

Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):1–8.

Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C, et al. Beyond adoption: a New Framework for Theorizing and evaluating nonadoption, abandonment, and challenges to the Scale-Up, Spread, and sustainability of Health and Care technologies. J Med Internet Res. 2017;19(11):e367.

Braun V, Clarke V. One size fits all? What counts as quality practice in (reflexive) thematic analysis? Qualitative Res Psychol. 2021;18(3):328–52.

Campbell KA, Orr E, Durepos P, Nguyen L, Li L, Whitmore C, et al. Reflexive thematic analysis for applied qualitative health research. Qualitative Rep. 2021;26(6):2011–28.

Google Scholar  

Banning M. A review of clinical decision making: models and current research. J Clin Nurs. 2008;17(2):187–95.

O’Neill ES, Dluhy NM, Chin E. Modelling novice clinical reasoning for a computerized decision support system. J Adv Nurs. 2005;49(1):68–77.

Arnold KF, Davies V, de Kamps M, Tennant PWG, Mbotwa J, Gilthorpe MS. Reflection on modern methods: generalized linear models for prognosis and intervention—theory, practice and implications for machine learning. Int J Epidemiol. 2020;49(6):2074–82.

Article   PubMed Central   Google Scholar  

Westerbeek L, Ploegmakers KJ, de Bruijn GJ, Linn AJ, van Weert JCM, Daams JG, et al. Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians: a systematic review. Int J Med Informatics. 2021;152:104506.

Henshall C, Marzano L, Smith K, Attenburrow MJ, Puntis S, Zlodre J, et al. A web-based clinical decision tool to support treatment decision-making in psychiatry: a pilot focus group study with clinicians, patients and carers. BMC Psychiatry. 2017;17(1):265.

Weingart SN, Simchowitz B, Shiman L, Brouillard D, Cyrulik A, Davis RB, et al. Clinicians’ assessments of electronic medication safety alerts in ambulatory care. Arch Intern Med. 2009;169(17):1627–32.

Baysari MT, Zheng WY, Van Dort B, Reid-Anderson H, Gronski M, Kenny E. A late attempt to involve end users in the design of medication-related alerts: Survey Study. J Med Internet Res. 2020;22(3):e14855.

Trafton J, Martins S, Michel M, Lewis E, Wang D, Combs A, et al. Evaluation of the acceptability and usability of a decision support system to encourage safe and effective use of opioid therapy for chronic, noncancer pain by primary care providers. Pain Med. 2010;11(4):575–85.

Wipfli R, Betrancourt M, Guardia A, Lovis C. A qualitative analysis of prescription activity and alert usage in a computerized physician order entry system. Stud Health Technol Inform. 2011;169:940–4.

PubMed   Google Scholar  

Cornu P, Steurbaut S, De Beukeleer M, Putman K, van de Velde R, Dupont AG. Physician’s expectations regarding prescribing clinical decision support systems in a Belgian hospital. Acta Clin Belg. 2014;69(3):157–64.

Ahearn MD, Kerr SJ. General practitioners’ perceptions of the pharmaceutical decision-support tools in their prescribing software. Med J Australia. 2003;179(1):34–7.

Swaby L, Shu P, Hind D, Sutherland K. The use of cognitive task analysis in clinical and health services research - a systematic review. Pilot Feasibility Stud. 2022;8(1):57.

Steyerberg EW. Applications of prediction models. In: Steyerberg EW, editor. Clinical prediction models. New York: Springer-; 2009. pp. 11–31.

Chapter   Google Scholar  

Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925–31.

Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. In: Doshi-Velez F, Fackler J, Jung K, Kale D, Ranganath R, Wallace B, Wiens J, editors. Proceedings of the 4th Machine Learning for Healthcare Conference; Proceedings of Machine Learning Research: PMLR; 2019;106:359–80.

Eini-Porat B, Amir O, Eytan D, Shalit U. Tell me something interesting: clinical utility of machine learning prediction models in the ICU. J Biomed Inform. 2022;132:104107.

Muralitharan S, Nelson W, Di S, McGillion M, Devereaux PJ, Barr NG, Petch J. Machine learning-based early warning systems for clinical deterioration: systematic scoping review. J Med Internet Res. 2021;23(2):e25187.

Rudin C. Stop Explaining Black Box Machine Learning Models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206–15.

Blythe R, Parsons R, Barnett AG, McPhail SM, White NM. Vital signs-based deterioration prediction model assumptions can lead to losses in prediction performance. J Clin Epidemiol. 2023;159:106–15.

Futoma J, Simons M, Panch T, Doshi-Velez F, Celi LA. The myth of generalisability in clinical research and machine learning in health care. Lancet Digit Health. 2020;2(9):e489–92.

Steyerberg EW, Uno H, Ioannidis JPA, van Calster B, Collaborators. Poor performance of clinical prediction models: the harm of commonly applied methods. J Clin Epidemiol. 2018;98:133–43.

Download references

Acknowledgements

We would like to thank the participants who made time in their busy clinical schedules to speak to us and offer their support in recruitment.

This work was supported by the Digital Health Cooperative Research Centre (“DHCRC”). DHCRC is funded under the Commonwealth’s Cooperative Research Centres (CRC) Program. SMM was supported by an NHMRC-administered fellowships (#1181138).

Author information

Authors and affiliations.

Australian Centre for Health Services Innovation and Centre for Healthcare Transformation, School of Public Health and Social Work, Faculty of Health, Queensland University of Technology, 60 Musk Ave, Kelvin Grove, Brisbane, QLD, 4059, Australia

Robin Blythe, Sundresan Naicker, Nicole White & Steven M McPhail

Princess Alexandra Hospital, Metro South Health, Woolloongabba, QLD, Australia

Raelene Donovan & Andrew McKelliget

Queensland Digital Health Centre, Faculty of Medicine, University of Queensland, Brisbane, QLD, Australia

Ian A. Scott

Digital Health and Informatics Directorate, Metro South Health, Woolloongabba, QLD, Australia

Ian A. Scott & Steven M McPhail

You can also search for this author in PubMed   Google Scholar

Contributions

RB: conceptualisation, data acquisition, analysis, interpretation, writing. SN: data acquisition, analysis, interpretation, writing. NW: interpretation, writing. RD: data acquisition, interpretation, writing. IS: data acquisition, analysis, interpretation, writing. AM: data acquisition, interpretation, writing. SM: conceptualisation, data acquisition, analysis, interpretation, writing. All authors have approved the submitted version and agree to be accountable for the integrity and accuracy of the work.

Corresponding author

Correspondence to Robin Blythe .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the Metro South Human Research Ethics Committee (HREC/2022/QMS/84205). Informed consent was obtained prior to interview scheduling, with all participants filling out a participant information and consent form. Consent forms were approved by the ethics committee. Participation was entirely voluntary, and could be withdrawn at any time. All responses were explicitly deemed confidential, with only the first two study authors and the participant privy to the research data. Interviews were then conducted in accordance with Metro South Health and Queensland University of Technology qualitative research regulations. For further information, please contact the corresponding author.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Blythe, R., Naicker, S., White, N. et al. Clinician perspectives and recommendations regarding design of clinical prediction models for deteriorating patients in acute care. BMC Med Inform Decis Mak 24 , 241 (2024). https://doi.org/10.1186/s12911-024-02647-4

Download citation

Received : 06 September 2023

Accepted : 23 August 2024

Published : 02 September 2024

DOI : https://doi.org/10.1186/s12911-024-02647-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical prediction models
• Clinical decision support systems
• Early warning score
• Clinical deterioration
• Clinical decision-making

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]